Merging Multiple Git Repos into One
In order to make life easier for our development team, we recently undertook the task of merging all of our separate application repositories into one, all inclusive repo. Logically, the previous structure looked like:
What we wanted was this structure:
Ideally, we’d find some way to preserve the histories of the files on trunk. We made the decision not to preserve the branches. This was not a big deal. The nature of the multi-repo approach meant that all branches were useless, if anyone bothered to branch at all.
Unifying these repos while preserving the history of the files on the respective trunks was pretty straight forward with some git magic and elbow grease.
Step 1: Create the Repo
Step 2: Import Existing Code
You’ll need to do the following two steps for each of the repositories you want to combine into the monolithic one.
These commands add the existing repository as a remote for your monolithic repo, then fetch the changes into that repo. The merge command takes whats in the existing repo and puts it on the master branch of monolithic repo.
While fetching the each additional remote, you will receive warnings about the lack of common commits. This is okay. The files do not have a shared history.
Step 3: Copy the merged files into the correct subdirectory
This is the bit that requires some elbow grease. When you merge the remote repo, the merged repo is going to have all the directories at the top level. Most likely you’ll want these in a sub directory. I used Finder to move the files I wanted.
After the files were in the correct subdirectory, I used the following git commands to properly add the changes so that the history won’t be lost.
The important line is git add -u which will mark all the removed files as renames when used in conjunction with git add <subdir_name>. This is the trick to maintaining history.
All done
You’ll need to repeat steps 2 and 3 for each of the repos you want to merge. Once you’ve finished, you should have one complete monolithic repo. Happy branching!
One Repo to Rule them All?
For the longest time our projects were setup in a mostly canonical maven way. Each module was separated into multiple sub modules. Each top level module got its own source-control repository.
The initial idea was each module was its own thing and our lives would be easier if we could version each project independently.
Reality did not match up with that thought. Our features and functionality grew along with the pace of our major releases. The once separated modules became developed concurrently as they call communicated with each other. To manage the maven tree, we have to either use SNAPSHOT everywhere or version everything. Each of those solutions have problems.
I’ve never had to manage this much source code changing this fast, so I could use some advice. I think the best solution would be to merge all the individual repositories into one repo. In my head, these are the pro’s and con’s I can think of:
Pro:
- All source code is managed globally across all projects, allowing for coherent feature branching
- With a guaranteed source tree layout, we can write tools to manage maven that will work for all developers
- A developer can get back to the state of any arbitrary build. This could be crucial for rebuilding production artifacts for a damaged system
Con:
- We can no longer tie specific modules to specific dependencies. Changing a core library will effect everything.
- We lose some access control. We may not want contractors working on core code, but only the Web front end.
- Similar to above, but developers will be forced to check out everything. With the wrong VCS, downloading branches and tags could be very expensive.
What I’d like to know is what else am I not thinking of? Has anyone else had some experience they’d like to share? Please leave a comment or send a tweet @vi_jedi.
Techcruch Talks about Scribit
Anthony Ha from Techcrunch interviewed our President Joe Fiveash about Scribit.
JQuery Line Ellipsis
I’ve been using the jQuery Ellipsis plugin on a few projects. However, on my latest one, I had the need to truncate text not on fixed heights, but a set number of lines.
I modified the plugin to do just that and threw it up on github. The Line Ellipsis plugin is still a work in progress, since detecting the size of a line is very unreliable. If you try it out and have problems, open a GitHub ticket. Or better yet, create a pull request.
Testing Like the TSA
David’s post perfectly describes my frustration with TDD, especially in long lived and complex projects.
Introducing Scribit
If you haven’t seen me in a while, its been because I’ve been so heads down in work. We’ve been pushing really hard over the last few weeks building out our new product Scribit. Scribit is a subscription service that gives any website access to high quality content from some of the top publishing brands.
A business has to sell something to make money. Content helps convince people that what you are selling is what they need to buy.
Lets say that you run a store selling craft beer. You want to tell people something about the beers that you offer. Before Scribit, you really only had two choices. The first is to write all that content yourself. Tasting beer is fun, writing about is a little less fun, but either way it takes a lot of time.
Of course, there are lots of sites that have good beer reviews. However they are all ad supported. Linking to one of those sites carries the risk that your visitor will see the ad for your competitor down the street and visit her shop instead.
Scribit allows you to take that content and put it on your site legally. If you’ve got Scribit, all you have to do is search for “craft beer.”
Once you’ve found an article you like, it’s a matter of deciding where on your site it appears and which social networks you want to publish to.
That’s it. The article is now in my twitter feed and on my site. It’s that easy.
We’ve put Scribit through the paces in a private beta period and now we’re opening it up for everyone. All new Scribit accounts start with a 30-day free trial. I also have a few complimentary subscriptions that I can hand out. DM me for details.
SXSW: Recap
SXSW has come and gone. Unlike all the live bloggers and active tweeters, I dropped into a social media hole and decided to just lump all my posts together from the safety and comfort of my Atlanta home.
A hot topic during the conference was the quality of the conference itself. From the reactions of people that had gone in the past, I was expecting something magical and compelling. In many ways, I was underwhelmed.
The panels were incredibly hit or miss. I had maybe two that I found very interesting, two more that were compelling, most were mediocre, and a few that were only memorable for being awful.
The entire event had the air of a Potemkin Village. All the special buildings and events put together by brands like American Express, Chevrolet, CNN, Turner, Nokia and Microsoft seemed to be erected to hide the lack of real substance in this year’s event. It’s not the brand’s fault. I think they are trying just has hard as anyone else to engage with their audiences. It seems that it’s more the matter of getting too large too fast.
The question of whether I would go again is remarkably complicated. Over the course of four days, I only had one great session and went to only one great party.
The biggest bright spot was all the interesting people I met. The joy of attending a cross-functional social conference is that you meet individuals who you wouldn’t normally come across. I talked to advertising reps, programmers, marketers, entrepreneurs, writers, financiers, social media managers and many more over the course of the event. That just doesn’t happen in the common industry focused events.
I felt pretty confident that I wouldn’t be back next year up until I got to the gate for the flight back to Atlanta. Out of the blue, someone recognized me from twitter. With him was some guy I randomly met in a restaurant nearly two years ago. Near us was a common colleague with a few of her friends, one of whom happened to sit next to me on the flight. Three reconnections and a new friend in what is normally a mundane event.
Looking back, that happened a lot over the course of the weekend. I think I’ll be back.
As for the panels (in quasi chronological order): Pitching Startups to Brands
Story Telling for Entrepreneurs
Interesting Companies that Had Beer
Real Time Data Changes Consumption Habits
SXSW: Data Visualization for Change
The amount of data available almost as an inverse relation to how actionable that information is. Proper representation is key for creating change.
The first panelist spoke to this topic and presented his workflow on how he creates visualization for large data sets. His best slide was one that showed all the iterations that went into his visualization for OECD’s life index. Responding to feed back is crucial for creating a representation that makes sense. In the interactive world, working with end user feed back is just as critical.
One fact they discovered through the usage of the form was people weren’t weighting each indicator as randomly as they had hoped. This reduced the effectiveness of the message the OECD is attempting to promote. A future revision is planned to account for this.
The second presenter was completely unprepared, showed a video, had technical difficulties and talked for far to long. It seemed like the appropriate time to call it a conference and head on out.
SXSW: The Secrets of Scaling Two Sided Markets
This was by far my favorite talk.
Two sided markets are everywhere. Two examples are bank cards and merchants, and content producers and content consumers. However, there are not any good frameworks for evaluating the optimal growth strategy for markets that exhibit this behavior.
The presenters set out to create an applicable model. The first step was analyzing the types of variables that would go into this model. Since they were approaching the problem of expanding a payments network, they chose to focus on merchants and card holders.
They settled on a few key metrics, such as cost of acquisition for a card holder, the cost of acquisition for a merchant, and the number of each side that would have to exposed to the service before converting to use the service. They modeled the transactional behavior of card holders and merchants with two power law graphs.
They decided to explore a few different strategies using this model over the course of a set number of rounds. The first three were one sided, focusing entirely on either consumers, low-volume merchants, or high-volume merchants. They also tried a few mixed strategies, alternating between consumers and low-volume merchants, or consumers and high-volume merchants.
With their example input, focusing on consumers was the correct course of action. However since the inputs are variable, one could run extra simulations to find the breaking point, or adjust it for real world feedback, such as a different acquisition cost.
Since so much of building a market is guestimation, it would be wonderful to have access to a configurable model such as this. If they make the code available, I will definitely be adding some R skill and playing with it.
SXSW: Kafka Didn’t Have Tumblr
I decided to drop into an arty session since it was near where I found parking. The topic was how the digital world is shaping how designers present things in the physical world an it’s implications.
The panel included a few artists that use art to question morality. These people generally annoy me and this session was no different. The single dumbest comment was what to the speaker was the very profound question of “if a google street view car sees something awful happening, will it stop?” Even she had the remarkable insight that cars have drivers. I get that the (Panopticon)[http://en.wikipedia.org/wiki/Panopticon] is a chilling concept, but the core of it is that some judgmental being is watching. I think this analogy breaks down with the sheer amount of image data being produced.
Other than the panopticists, the rest of the panel was entertaining, if not interesting. One presenter made a compelling comparison between the commercial imagery during the space race and what we see today. Those images look dated today precisely because they captured te public imagination at the time.
One presenter built his presentation around a sign that read “this restaurant is now close.” he was fascinated by a set of small musical segments that when played in shuffle mode created new music. I believe his concept is called a “composer.”
The final presenter and the organizer of the panel was the best. He brought into question how much of traditional philosophy and thought is applicable in this constantly connected environment. He related what is happening in the sharing-internet to what the scientist are doing at CERN with the LHC. Like pictures of cats or Tom Selleck with a waterfall and a sandwhich finding the Higgs Boson does not have any practical application today. However, that does not mean finding the Higgs Boson has no value. Likewise, this whole social sharing, micro-blogging, silly picture culture of today may seem pointless, but it may lead to a more compelling or interesting future.
