Category Archives:  Software and Services

Reviews pertaining to Software and Services

Me, My Photography, and SmugMug

Where I’m Coming From

I’ve been a long time PRO level subscriber as well as supporter of SmugMug. They are easy to use, have a great display, allows for unlimited uploads, and generally have a great support team that responds quickly.

I subscribed to the PRO level, now referred to as Business level, when it was $99.95. I did multi-year subscriptions to lock in my rate and for the last several years, I’ve been at that level. For what I was getting, it was a great deal, even though I’ve never been able to figure out, how they managed to make money, given the unlimited storage model they were working with. I had assumed they either made enough through commissions on sales or the bulk of their users were heavily under-utilizing the service.

Just before the recent Labor day weekend, I got an email from SmugMug indicating that effective mid-October, there would be a price bump for all Pro level subscribers of $100. So $99.95 subscribers get bumped up to the $200 mark and those at the $159.99 get bumped to the $25x level, etc. New subscribers get the $300 price tag.

For me, this would be a doubling of overhead for what is effectively an already expensive hobby. With cost of living going up, our household has been making cuts where we can. My hobbies and the subscription services tied to that hobby are no exception. The amortized monthly amount would effectively be going from $8.33 to around $16.66, so not a big monetary bump. But still, a bump.

Then, there is the question of business model. In SmugMug’s reply to the public and to their subscribers, the need for the price bump was to counter increased storage usage by PRO and BUSINESS users. The subscription price will go up, but nothing else will materially change. Those who don’t like the higher price can downgrade to the new PORTFOLIO account, which is $159.95/year. This really isn’t an option for those who still want to make monies off of their sales, as you lose the ability to set prices and make a profit from sales.

SmugMug did address the storage utilization issue and noted that mid-range PRO/BUSINESS users used at least 4X that of the lower tiers, and the heavy PRO/BUSINESS users used 4X of the mid-range PRO/BUSINESS users.

Instead of bringing in a tiered model, which would tie their storage utilization to a revenue stream, they decided on a fixed price bump. This potentially addresses the issue in a couple of ways:

  • Some users will leave, taking their storage footprint with them.
  • Users will pay more for their account, but might not increase their footprint, resulting in more net income.

However, here is the gotcha:

  • If users realize they aren’t getting anything from sales anyways, they might just downgrade to the $40/year account, leaving their large data footprint on the SmugMug servers.
  • The responses from SmugMug and the responses from end users might indicate that people may now make heavier use of the “unlimited” aspect of their SmugMug account.

What is the Cost of 1GB?

From what I can see from public posts and announcements, SmugMug’s storage capacity is handled by a combination of in-house storage systems and 3rd party back-ends like Amazon S3 cloud storage. Can’t speak to their in-house prices, but publicly available pricing via AWS S3 is around $0.15/GB, depending on region and durability. Inbound traffic to the AWS S3 is free, but outbound is billable.

My storage footprint at SmugMug is around 24GB, according to their metrics. That puts me at around $3.60/month, or $43.20/year. This is suspiciously close to the $40/year annual bottom tier subscription model with SmugMug, which still offers unlimited storage, but no customizable themes, domains, or custom pricing. Ie, they are front-ending your gallery.

Depending on the end-user’s equipment, 24GB is either ALOT of photos or it is only a moderate number of photos. For me, it represents around 2000 photos or around 12MB per image. I’ve got fairly large original JPEG(s) uploaded with the intention of printing them with the best detail possible. However, most will compress their images to under 2MB apiece. In which case, 24GB represents 12,000 images.

A PRO/BUSINESS user downgrading to an entry level account would still be able to upload as many images as they desired, just minus other niceties. If they have a front-end blog/gallery somewhere else, and are just using the SmugMug service to host images, then this can represent a problem for SmugMug. They effectively become a REALLY cheap unlimited image hosting service. Probably not a good spot to be in.

Where I’m Going

I’m not confident that SmugMug isn’t going to need to bump their prices again in the future. The price bump prompted my own re-evaluation of how I use their service, what features I use and what features I don’t use. My pain points and how a change can provide improvements to my own usage of my images.

I currently host my own blog website, this blog website, on my own virtual server. I _could_ host my images here, as that would incur a total of $0 additional cost to do so. This would also be the simplest means of hosting my images. However, the amount of storage space available to me is limited, and there are other considerations for that space usage: logs, temp files, other batch jobs running, etc.

So, looking to Amazon S3 storage hosting, I would incur a charge of around $3.60/month. However, that is only the case if I decide to store my original image files on Amazon S3. Since I’ll most likely no longer be printing the images, I no longer need to keep 12MB/image files and can bring that per image footprint down to around 2-4MB apiece. Or $0.60 – $1.20 / month to host my images. I would still be able to partner with 3rd party printing services like Fotomoto or BayPhoto using their ROES applications to do print orders.

Amazon S3 also has an interesting option, where you can associate a fully qualified hostname like say… gallery.wingtangwong.com to an S3 bucket. Replicating the folder structure of the images, file names, etc. would allow me to provide backwards-compatibility with already published URL links. No additional charge for this configuration.

Features like gallery passwords, gallery displays, and RSS/ATOM news feeds would need to be implemented with some coding either in WordPress or another mini server application. (TBD).

I have until mid-2013 to decide what to do, but in the meantime, I’m building out my plan B with the intention of migrating off of SmugMug. I would like to see SmugMug continue to grow, but they have made their business decision and it is pretty clear. In order to make their business model work, they will be focusing on photographers and subscribers who are actively marketing and promoting print sales through their site.

I’m not part of that demographic, and I don’t take it personally. Landscapes change and situations change. This is a change, and I’m changing my strategy and processes.

This has also prompted me to re-evaluate my photography in general.

Shrinking Down JPEGS! ( JPEGmini )

Recently, I came across this post from PetaPixel about JPEGmini

It details a company that has come up with a way to optimize an image so that it visually looks virtually identical to the original, but takes up a fraction of the original file size.That’s awesome! I even tested it against some image files:

Black Sea Nettles – JPEG shrunk using custom algorithm.
  • 24 megapixel JPEG / 12MB in size => 1.6MB resulting file after JPEGmini.
  • 12 megapixel JPEG / 4.8MB in size => 1.1MB resulting file after JPEGmini.
I performed a difference test against the original and the JPEGmini version. They are virtually identical. If you take the resulting difference image and adjust the levels, there are in fact different, but the difference can be attributed to minor random noise vs compression artifacts. Amazing!
DIY JPEG Compression Improvement
Being the kind of guy I am, I was wondering… how did they do this? According to the JPEGmini site:
  • improved algorithms
  • compressing to a level just before artifacting becomes an issue
Well, thinking that I could at least get some improvement, I leverage ImageMagick and some shell scripting.
I tested with the same images and came up with the following:
  • 24MP / 12MB => 2.8MB (1.7MB with updates below) (jpeg compression quality @73%)
  • 12MP / 4.6MB => 1.1MB (jpeg compression quality @87%)
The images produced were likewise virtually the same and a difference mask between the two in photoshop shows nothing, unless you auto-level. The amount of difference between my reduced images and JPEGmini’s reduced images were comparable. No new JPEG compression algorithm on my part, just applying basic JPEG compression guidelines from FileFormat’s JPEG Page:

The JPEG library supplied by the Independent JPEG Group uses a quality setting scale of 1 to 100. To find the optimal compression for an image using the JPEG library, follow these steps:

  • Encode the image using a quality setting of 75 (-Q 75). If you observe unacceptable defects in the image, increase the value, and re-encode the image. If the image quality is acceptable, decrease the setting until the image quality is barely acceptable. This will be the optimal quality setting for this image.  Repeat this process for every image you have (or just encode them all using a quality setting of 75).

The process that my script goes through:

  1. Compress original image at 99% image quality
  2. Do a comparison metric between the original and the new image.
  3. If the compression has resulted in a difference greater than a certain threshold, use the compression quality percentage prior to the current one.
  4. If difference is less than threshold, decrease image quality by 1% and repeat from step #2.
Using this process against the whole image, I could reduce <12MP files down to ranges similar to what JPEGmini was able to achieve, albeit slower. For files >12MP, I would achieve great savings in space, but not as great as JPEGmini, for a given threshold of difference.
Edit: Upon reviewing my notes, it looks like the threshold for <12MP and for >12MP is different from the JPEGmini files. When optimizing with the expanded threshold for larger files, I am able to get comparable to what JPEGmini gets.
How I Did The Compression Process
  • Use ImageMagick’s “compare” to generate “image distortion” readings:
  • convert -quality PERCENTAGE% original-image.jpg compressed-image.jpg
  • compare -verbose -metric mae original-image.jpg compressed-image.jpg
  • Using the example file from JPEGmini, establish a baseline:
    • 24MP image = ceiling of 370 points of distortion
    • 12MP image = ceiling of 265 points of distortion
  • Starting at 99% quality for JPEG compression, continue to reduce quality until the distortion measured exceeds the baseline. If it does, use the compression level from before.
    • The level of acceptable compression will differ from image to image. If the image has more pixels/images, then it will be able to contain more points of distortion before the image quality suffers.
    Optimizations (speed)
    To optimize the speed, especially for larger images, I scaled down the original image and performed the tests on the scaled down images. Once the ideal was located, I performed the compression on the original with the quality determined from the scaled down image.  For very large images, I take a crop from center of the image, which results in a more accurate quality prediction.
    Thoughts
    I can see why most applications don’t both with this process… it’s basically exhaustive testing to determine how much you can compress before image quality is degraded unacceptably. In addition to scaling down the image, I could also do quality percentage point skipping and backtracking when quality is degraded.
    What would be ideal is an API and library set that does an image quality analysis and gives you an optimally compressed JPEG out of the box. That is the value add of what JPEGmini is offering, and I think that it is a technology which any company involved with image storage should look into.
    The Code
    Note, the shell script/etc are not published, as the goal was to explore whether or not it could be done. Based on the description above, one can easily write the appropriate wrapper script or use the appropriate java/python/php/etc hooks.
    Update/Edited(8/29/2011):
    • Improved optimization to the quality locating logic resulted in a reduction from 5 minutes to 13 seconds to locate the quality setting and compress a 24MP image.
    • Planning on creating a Lightroom 3 export plugin for this scripted method.
    • Planning on trying to reduce the time required from 13 seconds down to under 10 seconds.

    Managing Resources – Part II

     

    This is a followup to my previous article, “Managing Resources – Part I“. It’s been half a year since I wrote that article and it has been a mixed blessing kind of trip. So, here are some of my thoughts from the attempt at managing my storage through complexity, and my thoughts going forward.

    How Did It Go?

    If you read part 1, you’ll note I went with a big box, lots of cpu cores, ram, and storage. I opted for complexity and multiple layers of abstraction to get all the features I wanted. I opted to get ZFS by emlpoying Nexenta and other Solaris derived solutions. Virtual machines to get the best of both worlds, and leveraging PCI-passthru to get a desktop experience on a virtualized server platform.

    In short, this did not work out very well. :(

    What Went Wrong

    From the start, there were problems:

    • PCI passthru only worked with a small subset of devices, and the devices it did work with were primarily storage controllers. Only specific, expensive, storage controllers. Display controllers were more or less out of the question, save as CUDA processor usages.
    • Without PCI passthrough for the storage controllers that I had, the option of using Nexenta in an ESXi VM with direct access to the underlying hardware to present storage options was a bust as well. Tests with Nexenta running on top of VMFS as presented through ESXi resulted in horrible performance numbers. Think something along the lines of 5MB/second read/write.
    • Juggling 10-12 1TB hard drives proved to be a nightmare in a conventional tower case. Likewise, the heat and noise generated by the drives alone made running the server 24/7 a problematic proposition.
    • The issues proved to be made worse by the fact that the OpenSolaris instance I was using had an advanced zpool version, which was not importable, at the time. This made importing and accessing the original data in my array a non-starter.
    Where Are Things Now
    In the end, I opted to get the  OWC Elite Quad Pro and load it up with 4 x 1TB drives, setting it up for Raid 5, and hooking it up to the Airport Extreme to use as a time machine drive for laptops and as personal dumping space, while the big server is being debugged.
    The main server is currently a mass of sata and power cables. I’ve added 2 quad port pcie x4 SATA cards to the server to handle more drives. I’m planning on re-imaging the box and loading it up with Linux(Ubuntu or Centos) and employing ZFSonLinux to implement ZFS storage pools and get RAIDZ2 up and running. This would remove the need for virtual machines, as I would get ZFS in Linux and would be able to run the applications I wanted to run, natively.
    I’ve also acquired 2 x 60GB SSD(s) to serve as L2ARC caching and ZIL logging for ZFS to improve the performance of the system. The intent is to have these two SSD(s) mirrored and having the mirrored device used for the caching and logging.
    What Needs To Get Done
    The remaining road block now is how to get the data off of the existing ZPOOL(solaris) and onto the new ZPOOL(linux). I’m thinking it will take the following form:
    1. Build new Linux environment / ZPOOL environment. (pool1)
    2. Export pool1 from Linux environment.
    3. Boot into old OpenSolaris environment and import all pools (pool0, pool1)
    4. While booted up in OpenSolaris, zfs send from pool0 to pool1.
    5. Once completed, export pool1 from OpenSolaris environment.
    6. Boot into Linux environment and import pool1.
    7. Confirm data is all good, then re-purpose all disks in the original pool0 OpenSolaris environment.
    The current capacity of the server is 14 SATA ports:
    • 6 on the motherboard :: mb0
    • 4 on quad sata 1  :: qs1
    • 4 on quad sata 2  :: qs2
    After taking into account the 8 disks for the existing array, this leaves 6 ports available for the new environment to be built off of. Building small RAIDZ groups, I’m planning on grouping the space as follows:
    • RAIDZ :: mb0:2 + qs1:0 + qs2:0    [ + 2TB ]
    • RAIDZ :: mb0:3 + qs1:1 + qs2:1    [ + 2TB ]
    This way, the six ports are spread across the motherboard and each of the 2 additional controllers. When the data is transferred over and the original 8 disks are freed up, they will be added to pool1 in a similar fashion:
    • RAIDZ :: mb0:4 + qs1:2 + qs2:2   [ + 2TB ]
    • RAIDZ :: mb0:5 + qs1:3 + qs2:3   [ + 2TB ]
    The remaining ports on the motherboard, MB0:0 and MB0:1 will be used for the SSD(s) for the L2ARC and the ZIL. As well as for the booting up of the Linux OS.
    The effective space will be 8TB of usable RAIDZ data. However, it is 8TB spread out over 12 x 1TB drives vs the original 7TB spread out over 8 x 1TB drives. The arrangement also ensures that should one controller card fail, the arrays will be recoverable.
    Upgrading storage and/or resilvering a replacement disk would also be significantly sped up by the fact that each RaidZ group is only comprised of 3 disks.
    If/when I decided to upgrade to 2TB disks, I can do so 3 at a time. Though ideally, they would all be RaidZ2 with perhaps more disks per group, but funds are no infinite. :( What this configuration buys me is more IOPS across the disks, and hopefully better performance.
    Future Upgrades & Potential Issues
    Once this is up and running, I may bring the OWC Elite unit into the mix via an ESATA controller, with the 4 drives in the unit presented as individual drives. However, because of the way that ZFS works, I would not be able to add a drive to each of the RAIDZ(s) since ZFS lacks the ability to reshape the geometry of the zpool as well as the ability to shrink a zpool by removing a device from it. So, future adding of storage would require ever more disks/space to enact reconfigurations, which makes using ZFS a bit problematic.
    One Avenue To Upgrade Via Adding More Disks, While Retaining Redundancy
    I suppose I _could_ swap out disks from each RAIDZ group and replace it with a disk from the external quad enclosure like so:
    From:
    • RAIDZ :: mb0:2 + qs1:0 + qs2:0   [ + 2TB ]
    • RAIDZ :: mb0:3 + qs1:1 + qs2:1   [ + 2TB ]
    • RAIDZ :: mb0:4 + qs1:2 + qs2:2   [ + 2TB ]
    • RAIDZ :: mb0:5 + qs1:3 + qs2:3   [ + 2TB ]
    To:
    • RAIDZ :: mb0:2 + es0:1 + qs2:0   [ + 2TB ]
    • RAIDZ :: mb0:3 + qs1:1 + es0:2   [ + 2TB ]
    • RAIDZ :: mb0:4 + qs1:2 + qs2:2   [ + 2TB ]
    • RAIDZ :: mb0:5 + qs1:3 + qs2:3   [ + 2TB ]
    • RAIDZ :: es0:0 + qs1:0  + qs2:1  [ +2TB ]
    • + 1 x 1TB hot spare : es0:3
    In this fashion, another 2TB of RAIDZ storage is added to the zpool, making it 10TB usable across 5 x RAIDZ blocks. The extra slot in the external SATA enclosure can be used to house a hot spare, to allow for auto-rebuilding.
    Basically, if you want to add additional physical storage devices… you will need to ensure that the devices are re-juggled around in such a way that the failure of any one device will not result in a failure of any RAIDZ component that makes up the zpool.
    In anycase, storage and data management continues to be a pain on the home front. :)