Tejus's Programming and startup blog

Access Denied Error when using XHR PUT and DELETE

Wed, 08 Feb 2012 08:22:41 -0500

Late one night, I was attempting to wire up a delete button on a Rails app using XHR. However, every time I attempted to make the XHR call, I saw AccessDenied in the server log and my session was un authenticated. Since it was late, I remapped it to a different URL and moved on.

However, the fact that this did not work still bothered me and when I revisited it after a good night’s sleep, the answer was quite obvious. My Ajax setup (copied from many Rails projects ago) looked like:

http://gist.github.com/1769263

I was only setting the X-CSRF-Token on a POST. Therefore, when the server received the DELETE verb, it killed the session, thinking that something was afoul.

Changing that line to:

http://gist.github.com/1769443

fixed the issue in the correct way.

Access Denied Error when using XHR PUT and DELETE

Wed, 08 Feb 2012 08:22:30 -0500

http://gist.github.com/1769263

I was only setting the X-CSRF-Token on a POST. Therefore, when the server received the DELETE verb, it killed the session, thinking that something was afoul.

Changing that line to:

http://gist.github.com/1769443

fixed the issue in the correct way.

@Autowire with AOP Proxy Classes in Spring

Thu, 05 Jan 2012 21:09:52 -0500

This is an issue that gets me once every blue moon. Lets say that I have two classes DependencyClass and MainClass:

http://gist.github.com/1568484

DependencyClass makes use of Spring’s @Transactional support, which is provided through AOP Proxies. This code works when I fire it up. However, if I add another implementation, and want to change the code to the following:

http://gist.github.com/1568539

I get this nasty on the application startup:

NoSuchBeanDefinitionException: No matching bean of type [DependencyClass] found for dependency

The reason for this is simple. Within the Spring container, the concrete class DependencyClass has been replaced by a proxy. Therefore, as far as the system is concerned, there is no matching bean of type DependencyClass.

Inevitably, when this occurs, I spend too much time looking to other issues with my application context setup. However, the solution is easy and straightforward, once you realize what the problem is:

http://gist.github.com/1568553

Basically, one needs to change the type back to an interface, then use named services and the @Qualifer annotation to inject the desired dependency.

Can Vertical Acuity Stop Your Site Losing Visitors?

Tue, 22 Nov 2011 09:58:39 -0500

Can Vertical Acuity Stop Your Site Losing Visitors?:

It’s always fun to wake up in the morning and see Mashable do a write up on your company.

Vertical Acuity Quick Pitch: Vertical Acuity is a self-serve content syndication…

(Trying to) Sanely Deal with Maven Dependency Versions

Sat, 05 Nov 2011 11:10:12 -0400

One challenge of the maven way is how to sensibly version dependencies in a large, but constantly changing, project.

In the ideal maven world, each module is a highly defined project, with a highly structured team and releases only occur once the contract has been fully defined, tested, and whatever else enterprise teams do. In many realities, projects are simply logical constructs that don’t have strict contracts between the other module. The production environment is a mix of highly stable and rigorously tested products and experimental, pushed-constantly projects.

In this environment, automatic versioning of dependencies is both unnecessary and costly, and the dependencies need to be frozen at branch time of the stable, top-level projects. The other requirement is that the changes to the poms need to be minimal, so that any merge conflicts can be easily resolved.

The approach I’ve used, combining a few different ideas from the internet, is to remove any version numbers referencing internal projects from child modules. The versions are then included as properties in the parent project.

Freezing the set of dependencies for the module tree is then a three part process of: - Change the version number in the parent pom - Change the version properties of the parent pom - Change the parent version number in the child poms

To illustrate how this works, consider a project that has com.tejusparikh.web with a parent pom com.tejusparikh.

http://gist.github.com/1341577

http://gist.github.com/1341587

If you wanted to freeze the poms at 1.0.RELEASE then the poms would look like the following:

http://gist.github.com/1341596

http://gist.github.com/1341597

The point is that every child pom would have a property as it’s version number, making it easy to manage an arbitrary number of children.

Of course, you don’t want to have to update everything manually. The only method I’ve found to effectively use maven is to wrap it all in rake. Therefore, I’ve created a rake task to recurse the directories and change the version numbers.

http://gist.github.com/1341631

Freezing all of your poms is as simple as calling rake mvn:version[/path/to/parent/dir,VERSION].

Getting Rid of My Personal Server

Wed, 19 Oct 2011 11:23:15 -0400

A few years ago, I stopped hosting my site on physical hardware and got a VPS from Linode. This worked really well for a few years, but I’ve gotten busy with other things and linux server administration really doesn’t interest me anymore.

Since I blog so infrequently, every time I was tempted to write a blog, Wordpress begged me to update it. By the time I installed the patches, updated the plugins, I blew the window and desire to write a blog post.

Therefore, through no real fault of Linode, I’m shutting down my VPS and moving to strictly to SaaS solutions.

The most important and challenging was moving the blogs to Tumblr. I wrote a small post for the tool I wrote.

The other challenge old SVN repositories off the machine. Since these are really just projects that I created years ago for very specific people and purposes, the cost of Github didn’t make sense. Instead I opted for the free plan on Bitbucket.

Finally, I am using Linode to handle some of my DNS. Since I moved my domains to namecheap, I just use their free domain service for most things. For anything that requires more, there’s always Amazon’s Route 53.

Just for kicks, I created my own “about-me” style page and tossed it out there with Amazon’s S3 for static websites.

Unless you are hosting a real application, there’s almost no reason to manage one’s own server. There are simply too many free or cheap alternatives that handle the administration overhead.

Using Noah to Configure Properties in your Spring Container

Fri, 07 Oct 2011 23:36:23 -0400

On a recent trip through our code base, I realized that there were multiple projects that used identical properties defined in files that sat on the file system. I find DRY is a superior pattern to copy and paste I wanted to consolidate these into a single location.

One approach would be to move them up a level in the source stack, and either rely on the properties being included in a jar or included via the build process. Both of these approaches suffer from some drawbacks and require rebuilds on a config change. Also the properties are still located all over the place, making it difficult to recover from mistakes or handle changes to the environment.

One of my co-workers, John Vincent, is currently developing Noah which is a lightweight node/service registry. However, there’s nothing that says you couldn’t use it as a centralized location for properties files.

I uploaded my properties file using Noah’s Ephemeral API. From there, it was a very trivial to make the properties available to Spring:

http://gist.github.com/1271788

However, I didn’t like the url hardcoded in the configuration file. I’d prefer to have them configured through some properties that do live on the filesystem. In effect, each server would just need to know where to go to configure themselves. This required more advanced Spring trickery. Since all PropertyPlaceholderConfigurer instances implement PriorityOrder, they will be implemented before any property expansion has become. Therefore, you have to create your own Ordered bean that wraps a PropertyPlaceholderConfigurer. The following code is adapted from this discussion forum.

http://gist.github.com/1271808

Then you configure it in your applicationContext.xml file like the following:

https://gist.github.com/1271812

Now, you just have to manage one file on the filesystem that points to where the rest of the application’s configs can be found.

Import your wordpress blog into Tumblr

Sun, 02 Oct 2011 10:50:35 -0400

Last week, I searched rather fruitlessly for a way to import my wordpress blog into Tumblr. There were a few snippets and examples out on the web, but they all used the older API or required email addresses and passwords for authentication.

Since I couldn’t find what I was looking for, I opened up my IDE, fought with OAuth, and created a Wordpress to Tumblr importer.

Using the importer is pretty easy, you just need ruby and bundlr. The script is a sinatra application so you should will only need normal Tumblr OAuth access and not require XAuth approval.

Re-evaluating Vaadin : A Difficult Framework for Building Web Applications

Thu, 09 Jun 2011 20:37:00 -0400

I just had to make the rather painful (well painful for my free time) decision to re-write the project that I and a team member had written in Vaadin with the old standby’s of JQuery, JSP’s, and Spring MVC. There was a lot of frustration that went into maintaining the project over the last year and it simply became very difficult to deliver what the product team wanted in a timely manner. The point of the post is not to bash the framework, but present the difficulties I encountered that made it unusable for us. I described the reasons we picked Vaadin in a previous post Vaadin. Many of the benefits still exist, but some of our core assumptions lead to major headaches down the line.

Integration with Spring

The vast majority of what we do and where we spend our time is on the server. We’ve built large libraries of internal functions that are exposed to other aspects of our application stack through Spring. We use Spring for security, talking to the database, configuring our remoting endpoints, and almost everything else. Unfortunately, you can’t dependency inject the Vaadin Application object.

The most sensible pattern is to create a static singleton that has the beans your application needs to access. However, this is very kludgy and somewhat defeats the purpose of dependency injection. Also, since the Application is one monolithic thing, it is impossible to make use of Spring Security filters unless you write the unauthenticated portions of your application outside of Vaadin or in another Vaadin container.

Another problem is most of the advanced DataContainers in Vaadin want to interact directly with the datasource. However, we have services in the middle that abstract access to those systems. Configuring Vaadin to accomplish this was more difficult than it needed to be, leaving the developer with two real choices, hack around Vaadin’s plugins or use basic containers that you initialize yourself.

Very Inconsistent Plugins

To give a specific example, one of the bigger challenges was integrating with the Lazy Query Container. Vaadin defines interfaces to adhere to, but many of them throw UnsupportedApplicationException, which makes it very difficult for the programmer to figure out what actually needs to be implemented for the desired behavior.

Odd Layout Behavior

When we started the project, we found layout to be very easy. Then we tried to make the look more like some of our existing products, which meant changing some of the default padding, margin, and sizes. All of this has to be done inside Vaadin or by overloading a Vaadin theme. Using normal stylesheets will often get you into trouble since it throws off Vaadin’s resize calculations.

In one concrete example, I added a Label with HTML content inside of a table. After making that change, there were nearly 30 empty rows at the bottom of that table. Getting rid of the Label made it work again. It took about 60 lines of Vaadin to accomplish what I had put together in 6 lines of HTML. Also Vaadin sets every DOM element to overflow: hidden so if one is not careful, all the elements will clip. Setting all the elements to have a different overflow value is not trivial due to…

Thousands of HTML elements

Open Firebug on even a simple-looking page in Vaadin and you will discover elements that are nested dozens of levels deep. Therefore, even changing something simple about the box model of a specific element involves changing the elements on the tree above it. Things that are taken for granted in today’s web world, such as adjusting the size of an element in Firebug, are impossible.

The raw number of elements has implications for the user experience as well. Vaadin depends heavily on javascript and javascript interpreters slow down with DOM size. Browsers also have issues rendering very large DOMs. One of our pages is a list of thumbnails with some meta information associated with them. We discovered that even on our Dual Core Macs with Google Chrome, the browser became unusable after 50 elements. Something like Google’s image search would not even be possible.

The other problem is that animations don’t work. The browser can’t re-render the page and move objects out of the way smoothly so any animation looks jerky and odd. I didn’t go through the effort of installing Google Chrome to get a IE6 experience.

Long Cycle Times

In a normal java webapp, you can deploy an exploded war and fix interface issues (JS, CSS, and JSPS) with a reload of a page and firebug. Since it’s impossible to adjust on the fly with firebug and everything in a Vaadin app is written in Java, even a 1px width change requires a rebuild/container restart. What was 3 minutes of firebug fiddling has just turned into 30 minutes of cycling. This would have been less of a problem had we stuck with only Vaadin supplied look and feel, but that is not always an option.

Widgets are Opaque

I had what I thought was a trivial requirement. I need to have a start date and an end date. The dates can only cover a 90-day window and the end date can only be 30 days after the start date. I wanted the date widget to disable any dates outside of this range. This turned out to be unsupported. I almost said “impossible,” but that’s not true. I spent about a day digging through Vaadin’s source to where it interacted with GWT, back to Vaadin’s source and so on until I discovered the small change I had to make to support this feature and wrote a custom widget. However, none of this was documented wasn’t part of the official API.

Whatever success I felt at finding the solution dissipated the when we attempted to upgrade the Vaadin version and discovered the the code that my new fancy date widget depended on had been removed and no substitue was readily available. Of course product didn’t want to drop the requirement, so we stuck with the old version with the working date widget.

Weird Problems with Statefulness

A Vaadin application is stateful, which has all sorts of implications. First, unless you explicitly write a “multi-windowed” application, you cannot have different sections open in different tabs. The state of one tab will update the other tab on a refresh. Also, it is incredibly easy to screw up your error handling so that the app never recovers and the user is forced to clear their cookies before they can use the application again. We’ve even encountered this with Vaadin generated restart dialogs.

The Straw that Broke the Camel’s Back

The point where I hit the “I can’t make this work anymore” button happened when we switched to using Vaadin through SSL. The problem with using Vaadin for authentication is everything is generated through Javascript, which renders the browser’s login manager useless.

Vaadin provides a solution to this problem with the LoginForm widget. Vaadin generates the action for this form not through a relative path, but by reading properties from the container. However, in our case, SSL is handled by NGINX and Jetty is blissfully unaware. This works 99.9% of the time, except for Vaadin’s LoginForm, which happily attempts to send the credentials over HTTP, while the rest of the application was HTTPS giving a very unfriendly dialog in Firefox.

Analysis

I think the primary problem with Vaadin and the challenges I experience was in picking something beyond what I was comfortable with for a relatively time-boxed project. I’m far more experienced with HTML, CSS, and Javascript than I am with Swing or any other thick frameworks.

Much of my early time with Vaadin was spent thinking back to how I used to do it and trying to reconcile it with how I’m used to doing it now. Vaadin is clearly not meant for developers that have even moderate web experience. Going with more native web technologies will be far easier and deliver better results. Vaadin may still be an option if you need a team of Swing (or maybe Android?) developers to build web applications, but I don’t have enough experience with those frameworks to pass judgement.

If you’ve had some experience with Vaadin, I’d love to hear your thoughts as well. It’s a framework that gets mentioned frequently among teams looking to deliver web solutions. Know where it has succeeded would be just as enlightening as me sharing my issues with it.

Logging Standard Exceptions with Spring AOP

Wed, 18 May 2011 20:55:07 -0400

One of the challenges of working on a large system with multiple developers is the management and handling of application errors and exceptions. Often, developers get frustrated with managing typed exceptions and all interfaces either throw or catch Exception. Another common approach is to change all exceptions to RuntimeException to avoid cluttering interfaces. An alternative approached is described in this paper by Andy Longshaw. His approach breaks exceptions into two distinct hierarchies, domain and system. Domain errors caused by logical faults in the application, such as a zero being passed as a divisor to a function. System errors are errors caused through the interaction with other components, like a network timeout when attempting to communicate with the database. Both exception types are checked exceptions, which means they must be caught and handled by each layer of the application. He presents guidelines on when and at which layer each type of exception should be logged. If you are using Java the obvious drawback of using checked exceptions everywhere is that each layer will be filled with code that looks like the following:

    try {
        service.doApiCall();
    } catch(DomainException e) {
        log.error("Domain exception occurred", e);
    } catch(SystemException e) {
        log.error("System exception occurred", e);
    }

The duplication of handling of api exceptions can easily be handled with a custom annotation and Spring AOP. The first step is to create the Boundary logging annotation. This annotation describes where exceptions should be logged.

    @Retention(value = RetentionPolicy.RUNTIME)
    @Target(value = {ElementType.METHOD})
    public @interface BoundaryLogger {
    }

Next, we need to create the interceptor. Along with just logging exceptions, I also choose to log the api calls. There are a few steps to creating an Aspect that can be used by Spring-AOP. First added the @Aspect annotation to the class declaration.

    @Service    // make this available as a spring bean
    @Aspect     // tell spring this class is an aspect
    public class BoundaryLoggingInterceptor {

Next, create a method that tells the Aspect to do something before the underlying method, annotated with @BoundaryLogger, is called. In this example, it’s to log the method invocation.

    @Before("@annotation(net.vijedi.springlogging.interceptor.BoundaryLogger)")
    public void logInvocation(JoinPoint jp) {

Similarly, triggering the aspect at method end can be handled with:

    @AfterReturning(
            pointcut = "@annotation(net.vijedi.springlogging.interceptor.BoundaryLogger)",
            returning = "retVal")
    public void logComplete(JoinPoint jp, Object retVal) {

The one we are especially interested in for this purposes of this blog post is exception logging. This is the entire method for logging exceptions:

    @AfterThrowing(
            pointcut = "@annotation(net.vijedi.springlogging.interceptor.BoundaryLogger)",
            throwing = "ex"
    )
    public void processException(JoinPoint jp, Throwable ex) throws SystemException, DomainException {
        if(ex instanceof SystemException) {
            // System exceptions were logged at source
            // do not log the exception, just the return
            logReturn(jp, getLog(jp));
            throw (SystemException) ex;
        } else if(ex instanceof DomainException) {
            logException(jp, ex);
            throw (DomainException) ex;
        } else {
            logException(jp, ex);
            throw new DomainException(ex);
        }

    }

The logException method is very simple:

    private void logException(JoinPoint jp, Throwable ex) {
        Log log = getLog(jp);
        log.error(ex.getMessage(), ex);
        logReturn(jp, log);
    }

How the logger is obtained is very important for accurate logging. Otherwise, it will appear that all logging messages are coming from the interceptor and not from the underlying method. Along with the confusion this could cause, it will also make it harder to use automatic log analyzers. However, it is very easy to get a logger in the context of the underlying class.

    protected Log getLog(JoinPoint jp) {
        return LogFactory.getLog(jp.getTarget().getClass());
    }

Once the Aspect has been created, the final step is to tell spring what to do when it encounters the annotation @BoundaryLogger. There are four steps, tell spring to use annotations to configure the environment, to scan the packages the annotations are in, and to create the aop environment.

This is all that’s need to use annotations for logging at the tier boundaries. The complete code for the example is available at my Spring Boundary Logging repo on Github.

Stripping HTML Tags from your ActiveModel Model

Tue, 29 Mar 2011 21:29:22 -0400

Finally a Rails post! This one is short, basic, and probably has a bunch of other examples on the net. Rails3 escapes HTML by default, so this isn’t strictly necessary, but I still believe that what goes into the datastore should be clean. After all, the data will probably last longer than the front-end. I found this post that explains how to do it for your ActiveRecord models. However, I don’t have columns. Instead I used the following before_filter in my model class.

    before_save :sanitze_html

    def sanitze_html
        @attributes.each_key do |attr|
            value = @attributes[attr]
            if(value.class == String)
                @attributes[attr] = strip_tags(value)
            end
        end
    end

It’s the same idea, but instead use the attribute map to pull the objects out. If it’s a String type, call strip_tags.

Data Serialization notes with Flex4 (Actionscript 3) and BlazeDS

Sat, 12 Feb 2011 14:10:47 -0500

Unfortunately, I’m back to developing Flex applications and mostly been focused on the communication layer between our Flex application and the Java backend. Adobe’s BlazeDS product is the easiest way to communicate between a Flex client and a Java backend. BlazeDS integrates cleanly with Spring, has annotations to expose methods as services and integrates directly with Spring security. However, like most Flex features, it has it’s quirks. Our model objects follow the Fluent Interface pattern and we make use of method chaining to remove some of the code clutter when setting multiple properties. Furthermore, our object models are interfaces since the representations may be pulled from a large variety of backend stores. Of course, this doesn’t work with the Blaze DS serialization. This post is about all the wrong ways to do this and the explanation for why it doesn’t work. I cover my solution at the end.

The (Simplified) Java Model

// net.vijedi.java.User 
interface User {
    String username();
}

// net.vijedi.java.HibernateUser
@Entity
class HibernateUser implements User {
    @Column
    private String username;
    
    public String getUsername() {
        return username;
    }
    
    public User setUsername(String name) {
        username = name;
        return this;
    }
}

Mapping to ActionScript in all the Wrong Ways

My first attempt was to map the AS object to the interface User

// net.vijedi.as.User

[RemoteObject(alias="net.vijedi.java.User")]
class User {
    public var username:String;
}

Inspecting the event.result returned in the Blaze DS callback showed that the framework had converted it to a AS User object. However, the username property was null. Changing the metadata to [RemoteObject(alias="net.vijedi.java.HibernateUser")] yielded the same result. Likewise, changing [RemoteObject] simply returned an empty ObjectProxy. The problem is in using the fluent interface pattern. Blaze DS requires the object to follow Java Bean schematics. This means that setters must not have a return type. Changing the net.vijedi.java.User interface to follow bean schematics didn’t have the desired effect. The ActionScript side showed an ObjectProxy with the correct values set. Blaze DS sends the concrete type information as part of the return. It will still create the object, but it will not be what you want. Since AS is not a duck-typed language event.result AS User will throw a runtime exception.

The Only Way to Do It

Changing the Java class to:

class HibernateUser implements User {
    private String username;
    
    public String getUsername() {
        return username;
    }
    
    public void setUsername(String name) {
        username = name;
    }
}

and the ActionScript class to:

[RemoteObject(alias="net.vijedi.java.HibernateUser")]
class User {
    public var username:String;
}

This actually worked. The end result is that if you would like to use Blaze DS’s mechanisms for object serialization, the best bet is to create a set of custom value objects that are used for communication. This prevents serialization failures resulting from changes to the underlying models and frees the rest of the interfaces from being tightly coupled with the front end.

One Last Note about Booleans

If you have a field named boolean isEnabled, most IDE’s will create the getter boolean isEnabled(){...}. However, Blaze DS doesn’t remap it back the same way and the Actionscript side needs to be var enabled:Boolean, without the “is.”

Reading about How Facebook Pushes Code

Mon, 17 Jan 2011 14:39:26 -0500

I was just reading this article found on Hacker News about the development process at facebook. The line I liked the most was:

most engineers are capable of writing bug-free code. it’s just that they don’t have an incentive to do so at most companies. when there’s a QA department, it’s easy to just throw it over to them to find the errors.

I think this ties back to another point, that engineers are responsible everything (front-end, back-end, database, design) on what they create. They own that slice of the product. Ownership is the key to getting people to act like responsible adults. When someone dumps a PSD, DB Schema, and a best practices doc on a developer, it’s too easy for the developer to pass the buck to someone else.

Messaging in Java

Sun, 16 Jan 2011 13:11:25 -0500

At my current employeer, we’re starting to reach the point where we need a messaging infrastructure to scale our product. The traditional Java approach is to use a JMS broker, such as ActiveMQ. However, there’s a new alternative API in the form of AMQP. Our first use-case is a fan-out queue and I took the opportunity to see how the solution would look using both API’s within Spring. You can find the code for my proof of concept on Github. I’ve only included the most relevant snippets in the post, so for a real understanding on how to configure everything, look through the source.

ActiveMQ

The heavyweight in open source Java Messaging is ActiveMQ. ActiveMQ is a traditional JMS solution. Spring provides a template class that abstracts the communication with the JMS provider. The template requires a JMS connection factory along with a destination name. The same connection factory can be used for both sending and receiving messages by classes in the same JVM. The relevant configuration bits are as follows:

This creates a destination queue, a connection factory with the host and the template for the producer. In this example, the consumer is configured as a MessageListener and therefore, does not need a template. The StatMessageConverter is a bean that converts the payload to and from JSON. I choose to send the messages as JSON since we have a multi-language infrastructure and JSON makes it easier for those diverse systems to communicate. With this configuration, sending a message with a JSON payload is as simple as:

    template.convertAndSend(message);

The consumer configuration is simpler. Since my consumer class implements MessageListener, I just need to create a listener-container and inject the bean:

Whenever the producer sends the message, the onMessage method of the consumer will be called by Spring.

AMQP

AMQP with Spring works much the same way. There is a template and a connection factory. However, AMQP is different than JMS. First AMQP is a wire-level protocol, whereas JMS is an API. Therefore, different AMQP implementations should be able to communicate with each other. Secondly, the paradigm is slightly different. In JMS, there is a queue and consumers and producers bind to that queue. In the AMQP world, producers talk to exchanges and consumers bind queues to those exchanges. Therefore the communication topology is fully within the configuration and is unknown to the code, making it trivial to go from a one-to-one exchange to a fan-out. The AMQP provider that made the most sense was RabbitMQ. RabbitMQ is a logical choice because it has first-party java libraries and is owned by VMWare, who also own Spring, which bodes well for the project’s future. Spring’s AMQP support requires a little more configuration compared to the JMS solution, but the basics are the same. You need connection factory, template, and a listener. The first difference is that there must be an administration bean:

Next the send template needs both the routing key and the exchange:

The statsExchange will create the exchange as part of it’s initialization. If the exchange already exists, it will bind to it. Next comes the creation of the Queue:

Since I want fan-out behavior, each consumer must have a unique queue name. Therefore, I use an instanceID instead of a configured property. Otherwise, the same rule for exchanges applies here, the consumers will just listen on the existing queue. The consumer is similar, but in this case we can use a regular Pojo. Since Spring’s AMQP support includes marshalling support for JSON, no special conversion needs to occur on the client side:

All that remains is binding the exchange to the queues:

Sending a message is identical to JMS:

template.convertAndSend(message);

Conclusions

This little exercise was interesting because it shows that from the programmer’s perspective, there is little difference between using AMQP or JMS. With a little care in interface design, the two solutions can be swapped with minimal development effort. For our needs, we decided to go with RabbitMQ. The solution just feels simpler for the different types of communication we need. Plus the use of an Erlang message broker feels like the right technology for the right problem.

Going Cable TV Free

Wed, 05 Jan 2011 22:19:25 -0500

The title of the post is just a little mis-leading. We’re not going cable free, we’ve been so for more than 3 years. I just felt compelled to write this post now because I finally have a setup that I like and is easy to use.

The first thing to know about going cable free is that it’s probably going to take a while to break even on cost. If you get a bundle, the cable bill is probably a relatively small portion, especially if you pass on premium packages. So if you want to save money, get rid of the sports packages and HBO.

Going solely to the web for entertainment is a tradeoff between cost and usefulness. There are plenty of devices in the $100-$150 range that provide some functionality. However, almost all of them have severe limitations. We watch movies from our hard-drives, dvds, sports from ESPN3, Netflix, Hulu and listen to music streamed through iTunes or Pandora. There’s no set-top box, short of a PC, that can do each of those services.

For simplicity of management, we stuck with Apple hardware, specifically older Mac Mini’s. Dual core Mac Mini’s are perfect for these applications because they have enough horsepower for most local HD content (up-to 720p) and have DVI and optical audio outputs, making connecting them to the home theater a snap. They are quiet as well. You can pick one up for around $300 on craigslist, which is more expensive than the dedicated set-top boxes, but it’s a real computer. We tried using linux machines, and while they were cheaper, it felt like I spent more time getting things to play rather than actually watching something. Since we made the switch to an Apple mono-culture, the setup has been rock solid.

The major downside of using computers is that the interfaces aren’t designed to be accessed effectively from a remote. You need a keyboard and mouse. We tried out a bunch of wireless input devices, both RF and Bluetooth. None of them have really worked well from the distance of our couch to our TV. The best solution we found for controlling the TV is to use our iPhones. We bought Mobile Mouse Pro which is available in the App Store for $1.99. Compared to outfitting each TV with a keyboard and mouse, it’s a bargain. Since it works over WiFi, we also don’t have any range issues beyond being close enough to see the screen.

There are a few free mouse applications available, but many of them didn’t provide full keyboard support. Mobile Mouse gives you access to function and control keys like escape. This makes it easy to fully do away with an attached keyboard. Mobile Mouse also reliably finds the servers on the network and wakes them from LAN, allowing me to set the boxes to go sleep when not used and save power.

Since almost all the video and audio devices are hooked up to some WiFi source, I can stream the same music through the entire house. Our Airport Express supports AirPlay, but I don’t always want to listen to what I have in iTunes. I would also like to stream to the other Macs. To accomplish this, I use Airfoil and have installed the Airfoil Speakers on all the machines. Then with a few clicks, I can send any audio from my desktop to anywhere in the house. Airfoil is a paid app and costs $25.

Finally, for controlling the desktop while I’m in another room, I use Remote HD on my iPad. Remote HD is $3.99 but is better than the free VNC clients. However, if I’m in the room with the machine, Mobile Mouse works better. It’s a bit pricey for the quality, but I’ll pay it since I don’t want to get out of bed in order to shut the music off.

That describes what we do to keep ourselves entertained without cable TV. I think we’re finally at the point where we saved money compared to going with cable, but it’s still really nice to have the power of your computer on your TV.

Macbook Air (13") Review

Mon, 06 Dec 2010 07:32:23 -0500

The question I had to ask myself was is it really worth upgrading from a 1st gen MBA with a 1.8Ghz Core 2 Duo to a brand new one with a 1.86Ghz Core 2 Duo and could I do something better with the money. Since I was already buying lots of gear for the upcoming trip, I put my old one up on CL and wandered over to MacMall. A few days later, I got rid of the old one, had the new one, and had a more manageable number in my bank account.

This machine has been nothing but fantastic and solves every one of the problems that I had with the original. The new machine runs much cooler. Even in a warm room, its comfortable to place on your lap. For someone that uses their laptop a lot on the couch, that alone is worth a price premium.

Because it’s cooler, it’s much faster. While the processor on the original was 1.8Ghz, the heat dissipation problems inevitably meant that one core would be shut down or you’d have to use a program like Coolerbook to keep it from overheating. A fully running processor coupled with 4 gigs of ram and a modern SSD makes for a very quick machine.

The last two points are more about how this version is not DOA. What makes the 13” the perfect traveling laptop is that along with the sleek looks and low weight, it’s really capable of doing everything you need. The extra resolution of the screen, up to 1440x900 from 1280x800 transforms picture management, programming, and video creation from an ordeal to possible. On this trip I’ve written some PHP, poked around a little with new search engines that are written in JAVA, edited videos of our house to show to our family that haven’t been to the states recently. And I did all that without pulling my hair out or getting frustrated at the computer. Sure, it’s clearly slower than my quad core desktop and has less storage that my wife’s 13” Macbook Pro, but it’s more than adequate for everything I need to do when I’m not near my desk or home.

Ruby Full Text Search Performance: Thinking Sphinx vs Sunspot Solr

Fri, 05 Nov 2010 09:25:03 -0400

Full text search support has come a long way since the early days of Ferret. I’ve been using Ultrasphinx for a few years, and while it runs great, it doesn’t work out of the box with Rails 3. Two search projects that seem to be garnering a lot of support from the community are Thinking Sphinx and Sunspot. Thinking Sphinx is the most logical successor to Ultrasphinx, since both utilize Sphinx as the search server. Sphinx works by reading information out of the database to build the search index. Communication with the Sphinx server occurs by sharing C “objects” over sockets. Sunspot uses Solr, a Java search server built on the Lucene search library. Sunspot communicates with Solr through its REST API, using XML. Although the search engine is written in Java, Sunspot bundles a version of Solr that runs as a standalone server to make deployment just as easy as Thinking Sphinx. Solr is a compelling alternative to Sphinx, since the most scalable Web apps (Facebook, Twitter) use Java behind the UI layer. Solr servers can be clustered and since they manage the index, Sunspot can automatically update the indexes when the model objects change. There’s no need to run a cron job to reindex the data or setup delta indexing like with Thinking Sphinx. However, the impact of XML serialization/deserialization required for communicating with Solr on performance worries me. Processing XML documents is not as fast as unpacking C objects. In order to test this difference, I created a little benchmark to measure the relative impact. The Readme describes the test in more detail along with providing the source and instructions so you can configure it to your needs. To give the test a slightly more realistic scenario, the benchmark was run within my Ubuntu VM while communicating with the search process running on my OSX host. The host box has four cores clocked at 2.66 GHz each. The Ubuntu VM had one core dedicated to it. There was plenty of ram available for both the search engine and the benchmark task processing the search results. Mostly I did this to ensure that Thinking Sphinx wasn’t cheating by using unix sockets for communication. I did 50,000 searches and printed out a timing after every 5000 searches. These are the results:

Runs    Thinking Sphinx       Sunspot
5000              38.49       1611.60
10000             38.54       1648.51
15000             39.06       1614.52
20000             38.86       1583.53
25000             39.78       1613.79
30000             38.83       1595.60
35000             38.34       1571.96
40000             38.06       1631.87
45000             37.57       1603.31
50000             38.23       1634.53
Total            385.80      16109.26

I had expected Thinking Sphinx to be faster, but not 45 times faster. Extrapolating the numbers out, one can run 200,000 searches in the time it takes Solr to run 5,000. This was just a rough test to see the relative difference and is purely based on read performance of a few hundred records. It’s possible that proper tuning could improve performance or frequent re-indexing could degrade Thinking Sphinx’s performance, but it’s hard to see that chasm closing enough for there to be comparable performance when the search index can fit on one machine.

Evaluating Vaadin: A Java Web Application Framework

Thu, 05 Aug 2010 15:54:53 -0400

Vaadin is an interesting RIA platform built by Vaadin, LTD. Vaadin differs from your standard RIA by existing almost exclusively on the server. All application state remains on the server and all events are handled through communication with the server. The display layer is written as a layer above GWT and the whole project is open source and free. This evaluation post is the first in a series of my lessons learned while evaluating this framework. Vaadin is a good fit for all the web-applications that fit in the general bucket of “enterprise support tools.” The reasons why are covered after the jump.

What’s special about it?

I think the biggest departure from the frameworks that I’ve been using for web programming is that Vaadin is stateful. The client state is tied in the session with a state on the server. This has some major implications for deployment, such as requiring sticky-sessions in load-balancing and shared caches for failover. The developers of Vaadin made the framework stateful to facilitate building all the interaction code on server. The browser is only used for display. This means that you can use your ordinary Java toolset to walk through your event handling. You don’t need firebug or any javascript knowledge, unless you are creating a custom widget. The Book of Vaadin provides lots of detail on this model in their chapter on Architecture. I have no desire to repeat the documentation here, but just provide enough of a glimpse in case you are more interested.

Would I use it?

I think Vaadin fits in nicely within a specific niche. I would not use it for a normal website (nor do they recommend it), nor would I use it for a consumer-facing or heavily trafficked web application. The stateful model just doesn’t scale like that. However, there’s a whole other class of applications, management consoles, intranet web-apps and other systems that could just live on one server that are a perfect fit for Vaadin. Even more so if you can control which browsers the users run (IE javascript is really slow). These are the things that I really liked about the product:

Complete documentation. You don’t have to dig through a user-generated wiki. They have the whole book up on the website.
Friendly forums. A search shows multiple people that had the same problem. Instead of the usual, “use the search function,” that you see on a lot of other forums, there were real replies.
All java. A developer can build an entire web-application with just Java. That’s really important if web-application programming isn’t a core competency.
Doing things the Vaadin way is really easy. For instance, the hbncontainer makes building an app around hibernate trivial.
The widgets really work like their desktop equivalents. The table responds to keystrokes and the split panels work.
Vaadin hides the complexities of layouts on the web. You just specify your container strategy.

Were there problems?

While Vaadin is pretty easy and built on a language I’m familiar with, there was still a learning curve. Most of my issues came from doing things outside of the Vaadin way.

Vaadin and Maven don’t seem to get along too well. Trying to install the Visualizations plugin was a nightmare. Eventually I had to install it and its dependent sources into my test project to build the widget set.
Documentation and examples on how to leverage your existing DAO layers was sparse and incomplete.
The same goes for Spring integration, esp Spring Security.

Conclusion

If had to build a web app and didn’t have a web team, Vaadin is good way to go. You can build apps that have more than the basic “developer” look and feel without a whole lot of effort. The documentation lapses will surely disappear as the framework gains steam. The next post I’m going to write is about how to use your existing DAO structure with Vaadin.

Creating a Custom XSLT Function in Saxon HE

Fri, 23 Jul 2010 09:31:13 -0400

In our XSTL workflow we make use of a lot of XPATH 2.0 features, such as it’s built-in regex support. Unfortunately, the default Java6 XML parsers only support XPATH 1.0. The library we settled on is Saxon HE, since it was free, supported the features we needed, and could be extended with Java functions. One of my tasks was to convert all relative paths in an XHTML document to absolute paths. The server prefix was set as variable in the stylesheet. The transformer would have to determine if the selected path is a relative url, then work to resolve what the absolute path is based on the root passed into the page. It could be done with an advanced XSTL template, but since we already had the resolution function written in Java, it made more sense to write a Java plugin to Saxon. One of the missing features of Saxon HE is the seamless, reflection based integration of plugins. However, one can use the Extension Function API to achieve the same results. First on the agenda is creating an extension point:

package net.vijedi.saxon.extensions;

import net.sf.saxon.expr.StaticProperty;
import net.sf.saxon.expr.XPathContext;
import net.sf.saxon.functions.ExtensionFunctionCall;
import net.sf.saxon.functions.ExtensionFunctionDefinition;
import net.sf.saxon.om.*;
import net.sf.saxon.trans.XPathException;
import net.sf.saxon.value.SequenceType;
import net.sf.saxon.value.StringValue;

public class AbsolutizeUrl extends ExtensionFunctionDefinition {
    /**
     * The function will need a name you can call
     */
    private static final StructuredQName qName =
            new StructuredQName("", 
                    "http://vijedi.net/", 
                    "absolutizeUrl");

    @Override
    public StructuredQName getFunctionQName() {
        return qName; 
    }
}

The extension point extends ExtensionFunctionDefinition. I went ahead and created a constant that will store the name of the class and the function to return it. You will use this to access the function from inside of your XSLT. Now it’s time to think about the interface of this function. The function can take up to two string parameters, the absolute url base, and an optional url to process. The url to process is optional since it is not a requirement that an will have an href attribute. The function will return either a string or null if the second parameter does not exist. This is how you define this interface in the code.

private final static SequenceType[] argumentTypes = new SequenceType[] {
        SequenceType.SINGLE_STRING,
        SequenceType.OPTIONAL_STRING
};

@Override
public int getMinimumNumberOfArguments() {
    return 1;
}

@Override
public int getMaximumNumberOfArguments() {
    return 2;
}

@Override
public SequenceType[] getArgumentTypes() {
    return argumentTypes;  
}

@Override
public SequenceType getResultType(SequenceType[] suppliedArgumentTypes) {
    return SequenceType.makeSequenceType(
            suppliedArgumentTypes[0].getPrimaryType(), StaticProperty.ALLOWS_ZERO_OR_ONE);
}

Once the interface is defined, it’s time to define the actual work. The actual call is handled by a class that extends ExtensionFunctionCall. I like to define these as inner classes of the ExtensionFunctionDefinition. The pattern for this class is pretty simple. You need to process the arguments to the function. Saxon will give you wrapped arguments that you will need to unwrap. Then you need to call the actual logic (which should be in a separate class for re-usability) and finally wrap and return the value. Just as crucially, you need to override the function that tells the Saxon parser to use your implementation of ExtensionFunctionCall for this definition.

    @Override
    public ExtensionFunctionCall makeCallExpression() {
        return new AbsolutizeUrlCall(); 
    }

    private static class AbsolutizeUrlCall extends ExtensionFunctionCall {

        @Override
        public SequenceIterator call(SequenceIterator[] arguments, XPathContext xPathContext) throws XPathException {

            StringValue pageUrlSV = (StringValue) arguments[0].next();
            if(null == pageUrlSV) {
                return EmptyIterator.getInstance();
            }

            StringValue hrefUrlSV = null;
            if(arguments.length > 1) {
                hrefUrlSV = (StringValue) arguments[1].next();
                if(null == hrefUrlSV) {
                    return EmptyIterator.getInstance();
                }
            }
            
            String pageUrl = pageUrlSV.getStringValue();
            String hrefUrl = hrefUrlSV.getStringValue();

            // Url transformation magic goes here

            Item item = new StringValue(fullUrl);
            return SingletonIterator.makeIterator(item);  
        }
    }

That completes the definition of the function. You can find the full example code on GitHub. Now that you’ve written an extension, you need to tell Saxon that this function exists. For this, you will need to add the following to wherever you are currently accessing the TransformerFactory.

private TransformerFactory getTransformerFactory() throws net.sf.saxon.trans.XPathException {
    TransformerFactory tFactory = TransformerFactory.newInstance();
    if(tFactory instanceof TransformerFactoryImpl) {
        TransformerFactoryImpl tFactoryImpl = (TransformerFactoryImpl) tFactory;
        net.sf.saxon.Configuration saxonConfig = tFactoryImpl.getConfiguration();
        saxonConfig.registerExtensionFunction(new AbsolutizeUrl());
    }
    return tFactory;
}

This code checks to see whether or not you’re using a Saxon processor, and if so, registers your new function within it. Finally, it’s time to update the stylesheet to use the new function. You’ll need to add the function to the namespace, using the same parameter found in the second argument of the StructuredQName constructor.

Now you can use it like any other function:

Keeping Test Emails out of the Wild In Rails

Tue, 29 Jun 2010 23:24:47 -0400

There are a few cardinal sins for a developer: deleting the production database, deploying code to the wrong machine, and sending out emails to all the mock users. These situations happen because the terminal window on production looks an awful lot like the terminal window on dev. I don’t have solutions to the first two problems, but preventing emails to the test user database is pretty easy in Ruyb on Rails. Two solutions I’ve found are santize_email and mail_safe. These are both ActiveRecord extensions that allow the user to set an override address that will be the recipient of the email. Sanitize_email is the first solution I tried. It’s configuration is straight forward and it can either be installed as a plugin or gem. Personally, I like gems because they can be shared across multiple projects. Once you’ve installed the gem, you need to configure an initializer with the following (from their README):

    # Settings for sanitize_email gem.  These can be overridden in individual config/%env%/environment.rb files.

    require 'sanitize_email'
    ActionMailer::Base.sanitized_recipients = "jtrupiano@gmail.com"
    ActionMailer::Base.sanitized_bcc = nil
    ActionMailer::Base.sanitized_cc = nil

    # optionally, you can configure sanitize_email to to include the "real" email address as the 'user name' of the
    # "sanitized" email (e.g. "real@address.com ")
    ActionMailer::Base.use_actual_email_as_sanitized_user_name = true # defaults to false

    # These are the environments whose outgoing email BCC, CC and recipients fields will be overridden!
    # All environments not listed will be treated as normal.
    ActionMailer::Base.local_environments = %w( development test staging )

You can set the configuration to override the recipients, the cc, and the bcc. One could use sanitize_email to automatically set a bcc for all production emails, along with setting all three to prevent emails from going into the wild. Another neat feature is that it provides all Mailers with the force_sanitize method, which programmatically traps emails through a specific path. Mail_safe is available strictly as a gem. It’s configuration is a little simpler. The following is the basic config from their README:

  if defined?(MailSafe::Config)
    MailSafe::Config.internal_address_definition = /.*@my-domain\.com/i
    MailSafe::Config.replacement_address = 'me@my-domain.com'
  end

It doesn’t allow distinct addresses for bcc, cc, and recipients, preventing it’s use for automatic override addresses. While it lacks that feature, it does allow addresses for white-listed domains to be delivered, as well as allowing users to provide a proc for the options settings. For example:

    MailSafe::Config.internal_address_definition = lambda { |address|
      address =~ /.*@domain1\.com/i ||
      address =~ /.*@domain2\.com/i ||
      address == 'full-address@domain.com'
    }

    # Useful if your mail server allows + dynamic email addresses like gmail.
    MailSafe::Config.replacement_address = lambda { |address| "my-address+#{address.gsub(/[\w\-.]/, '_')}@gmail.com" }

Our choice was to use mail_safe since it did allow white-listed domains to be delivered normally. This fit the feature set of giving us a safety net incase our mock data got pushed to a system with a live mail-server, while still enabling our test users to use the application.