Vladimir Vuksan's blog

Velocity 2010 was an excellent conference. Following are my takeways from the conference. There is tons more but following are some of the things that made a good impression and are likely not hard to do

Web performance optimization

• Look at your Javascript. That is one of the major reasons for page slowness since Javascript download and parsing blocks other elements of the page to be loaded. Use Google's Closure Compiler to optimize Javascript by merging code and eliminating duplicate or unneeded functions. Check out Anne Sullivan's Progressive Enhancement slides.
• Yahoo release Boomerang which they describe as a piece of javascript that you add to your web pages, where it measures the performance of your website from your end user's point of view. It has the ability to send this data back to your server for further analysis.More details at http://github.com/yahoo/boomerang
• Version your CSS/Javascript and set expire times for 10+ years. Check out slide 24 from Theo Schlossnagle Scalable Internet Architectures slides.
• Use Cookies as a "distributed database". If you are concerned about security or tampering encrypt the cookies.
• Use JQuery sparingly. It takes 200-300 ms to parse it. This is even worse in the mobile world.
• Google rewrote their show_ads.js with ASWIFT which causes script loading to be asynchronous and not block other elements from loading. More about "Don't Let Third Parties slow you down" and  http://www.royans.net/arch/speeding-up-3rd-party-widgets-using-iframes/

Mobile performance optimization

Most of the recommendations have been taken off Maximiliano Firtman's Mobile Web High Performance. You can view slides here.

• Avoid JQuery unless you really need it. Check out slide 90. It takes 1.8 seconds on iPhone and 4 seconds on Android to download and parse JQuery. Use mobile optimized frameworks such as baseJS and XUI
• Avoid DNS lookups and minimize number of requests since they are slow
• Embed CSS and Javascript on the home page. After onload download external CSS and JS.
• Use inline images (slide 56) and pictograms
• Avoid redirects
• Use native constructs especially for Webkit browsers e.g. -webkit-text-stroke
• Keynote announced their Mobile Testing tool for desktops that looks promising http://mite.keynote.com/

SSL/Security

• According to Google SSL overhead these days is pretty minimal. Around 1% on today's servers.
• Pet peeve about the presentation is they were advising everyone to use less secure key lengths ie. 1024 bits and RC4 cipher to improve performance. It is true that adding SSL to insecure connections is certainly an improvement but it should be qualified. E-mail probably fine. Financial sites probably bad.

Scalability

• Hidden Scalability Gotchas in Memcached and Friends by Neil Gunther (author of Guerilla Capacity Planning) and Shanti Subramanyam discussed their findings around memcached. They used quantitative analysis to analyze different memcache versions. Based on their analysis using Neil's model memcache 1.4.5 has higher contention than 1.2.8.

Culture

• Thoroughly enjoyed John Rauser's Creating Cultural Change

At a previous job for availability and business continuity reasons we set up a geographically redundant data center because even the best data centers will have outages. No matter what a vendor tells you processes are never followed fully. You can also have a major disaster with critical pieces of your hardware that may cripple or disable your whole infrastructure ie. switch goes crazy etc.

Service we provided was critical so highest availability was imperative. Management wanted an active-active set up ie. use both data centers in a load-balanced fashion however that would have entailed extensive application rewrite due to the nature of our application and the level of database transactions involved. Thus we settled on a hot-cold configuration where we would have an active site that was serving customers and a cold site that was kept up to date via replication. In case of trouble (as determined by ops) we would fail-over our hot site to the cold site. This is fairly straight forward except for the part where you are actually failing things over ie. your hot site is down, you break off replication, change DNS entries, start up all the necessary services however due to DNS caching some of your customers are still pointing to your "dead" site. Depending on your browser this could be 30 minutes+. Did I mention this service was critical ?

We went through the list of possible options on how to resolve this

1. Use an outside party load balancer(s) ie. an off-site load balancer(s) that would proxy traffic to the site that was live. This seemed like a plausible idea however we didn't like the fact we were introducing yet another failure point and adding latency due to extra round-trip.

2. Changed DNS TTL to 2 minutes however that was also insufficient due to different browsers behavior. For example IE 6 (perhaps even higher) will cache DNS entries for 30 minutes

http://support.microsoft.com/kb/263558

3. Use round-robin DNS aka. multiple DNS A records with a "twist"

What we did there is put both of our data center's IPs into the A record for our site ie.

www.domain.com   IN A 1.2.3.4
www.domain.com   IN A 9.8.7.6

What happens with most browsers is that they will attempt the first IP and if they get a connection refused they will try the next (and next if you have more than 2). This actually works quite well e.g. even if the browser was getting requests from 1.2.3.4 if 1.2.3.4 all of the sudden goes down it will in sub-second time fail-over to 9.8.7.6. The "twist" we added was that we only answered on the active colo IP and returned connection closed on the inactive. If we needed to failover we'd just swap one colo and deactivate the other. Quick failovers here we come .

This all worked great for some time until we started receiving isolated reports that people weren't able to access our site. Investigating the issue further we discovered that all of the people having connectivity issues were behind a transparent HTTP proxy. In this particular case the transparent proxy would not return connection refused but "page not found" or something similar neutralizing our clever hack .

Obviously if you audience is different and you know your users don't use proxies you could use this approach however this doomed it for us.

If you need a quick way to determine when a certain SSL certificate expires you can utilize following approaches. In both examples server I am trying to check is called webserver.domain.com.

If you have Nagios plugins installed you could type

# /usr/lib/nagios/plugins/check_http -p 443 -S -C 15 webserver.domain.com
CRITICAL - Certificate expired on 11/01/2009 11:23.

That's easy. However what if you don't have Nagios plugins. In that case you can do the same with OpenSSL and s_client. Look for notAfter field.

# echo | openssl s_client -connect webserver.domain.com:443 | openssl x509 -noout -dates
...
notBefore=Nov  1 11:23:30 2008 GMT
notAfter=Nov  1 11:23:30 2009 GMT

Easy .

This is a common problem across many different platforms. You generate a CSR, get a certificate but forget or don't realize that besides installing the signed certificate you need to install the CA (Certificate Authority) Intermediate certificate. As a result some of the older browsers may complain about an invalid certificate or Java code will fail with following error message

Exception in thread "main" javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: CA key usage check failed: keyCertSign bit is not set

Solution is to download the intermediate certificate from the CA e.g. Verisign and GoDaddy and include it with the certificate. For instance in Apache you need to include SSLCertificateChainFile directive with the path to the intermediate certificate. On Cisco loadbalancer you would need to use following Cisco document. Specifically this directive

ACE-1/routed(config)# crypto chaingroup intermed-1
ACE-1/routed(config-chaingroup)# cert intermediate.pem

The chaingroup needs to be applied to the ssl-proxy service in addition to the already configured certificate and key.

ACE-1/routed(config)# ssl-proxy service proxy-1
ACE-1/routed(config-ssl-proxy)# chaingroup intermed-1

If you got your certificate from Verisign you can check whether you installed it properly here

https://knowledge.verisign.com/support/ssl-certificates-support/index?page=content&actp=CROSSLINK&id=ar1130

You can always of course misconfigure things causing lots of time to be wasted. For instance on one occasion a well known managed hosting provider that was in charge of configuring the Cisco load balancer configured the load balancer as follows

crypto chaingroup some.domain.com
   cert some.domain.com.cert
   cert intermediate.cert
ssl-proxy service vip-1.2.3.4-ps
   key some.domain-com.key
   cert some.domain.com.cert
   chaingroup some.domain.com
   ssl advanced-options ssl-parameter-map-1

This is incorrect as the server certificate SHOULD NOT be included in the intermediate certificate chain. Otherwise the helpful Verisign test applet will complain with following message.

Two certificates were found with the same common name. The certificate installation checker cannot determine which is the correct certificate for the Web server. Remove the incorrect certificate and then test again.

Most browsers will work correctly however Java code will exhibit errors from the top of the article. Solution for the above problem is this

crypto chaingroup verisign-intermediate
  cert intermediate.cert

Then included that chaingroup in the ssl-proxy directive. Once that was done the issue went away. Hope this saves someone some debugging time.