By Trevor Baca, VP Software Engineering


How do you keep old code from breaking when you write new code?

The answer is, and has always been, the running of a regression battery of tests. If you have custom classes in your code that, for example, are supposed to allow for multiplication by positive numbers and zero but not by negative or imaginary numbers, then writing a couple of tests to make sure that multiplication by positive numbers and zero work is the way to go. Even better, include a couple of tests to make sure that multiplication by negatives, imaginaries and other types of number don't. As you write more code, write more tests. The tests together constitute your regression battery. Then run your regression battery and make sure your tests all pass at important times. Such as before any major or minor release. And preferably at lot more frequently than that, such as before committing new code to the repository. A good regression battery is a requirement for a good team.

And now a small distinction in the times at which developers write new tests for inclusion in the battery is having big consequences in the world of software development. Enter the notion of test-driven development.

Test-driven development (TDD) helps us develop reliable code faster by putting testing at the center of the process. We've been migrating more and more of our development at Jaduka and NetworkIP to a TDD way of doing things. And the results have been exceptional.

So what is TDD? Take a look at this recent post from molecular biologist and Python enthusiast Titus Brown.

TDD asks that we write our tests as we go rather than at the end of the process. Given that almost all recent software development models are iterative and incremental, this makes good sense. How often does testing get squeezed in at the very end of a project? Too often, of course. And authoring test cases as we go can help considerably.

Supporting the different Jaduka APIs requires an approach to development that prizes consistency. Once any API vendor releases new methods, the public interface to those methods needs to stay pretty much the same. Because users like it that way.

But working on the back-end systems and networking code that makes Jaduka run requires a different set of tools. And a very important one of those is turning out to be TDD.

By Trevor Baca, VP Software Engineering

Check out this post Ben Collins-Sussman posted to his blog on Thursday. Ben's one of the primary authors of Subversion -- the repository we use for all our projects around Jaduka and NetworkIP -- and has been at Google for some time now as a lead engineer.

Ben's post addresses what he identifies as "programmer insecurity" -- why do we as developers always wanna hold off on releasing our own code until we've reached some future state of perfection ... and then why do we wanna cover our tracks when we're done? -- as an entry point to talk about why programmers work the way we do. And also how the tools that we chose to work with can help us hide or share our results.

It's a point well taken: there's no question that "commit early, commit often" helps foster a solid team understanding of what the key subsystems in the codebase are actually *doing*. And this seems to be just as much the case with projects internal to our own teams here as it is with open source initiatives.

I've blogged elsewhere about the value of different tools -- like UNIX screen and conference bridges -- for developer collaboration. Ben's post makes the additional point that, above and beyond our favorite collaborative tools, we have to work to ensure a collaborative workflow on our teams.

Well worth checking out.

By Trevor Baca, VP Software Engineering


If you haven't already heard, click over to Google's announcement of the new app engine development and runtime environment.

Summary: write a web app and deploy to Google's massive array of servers. APIs exist for user authentication, data storage, mail, URL fetching, etc. 500MB of storage and CPU bandwidth for 5 million pageviews a month. Only free accounts at the moment with the possibility to buy more computing or storage resource coming in future.

The interesting part? Google app engine is 100% python. (For now, anyway.) The web framework is Django (which is to python what Rails is to ruby). And you have to write all application code in python (as opposed to PHP or ruby).

Oh, and Google announced the app engine last night with 10,000 accounts available. Which are now all taken. Every last one.

It makes sense. Guido van Rossum, python's inventor, now works at Google. And python is widely reported as the single most important language in the Google's running and management of its network and absolutely enormous arrays of servers.

We're just now finishing up a three-year migration of the most important parts of the Jaduka and NetworkIP codebases from C to python. The migration has been challenging -- all migrations are -- but extremely beneficial. We're realized a 10-to-1 compression of lines of code, far more flexible database access, and -- most importantly of all -- dramatically faster implementation of every realtime telephony service we've put our hands on since then.

But full disclosure -- our use (and love) of python extends only to our core realtime systems code. We manage a complex network of telephony switches, routers, and application-, database- and statistical process control-servers all in python. But our web applications?

They're all in PHP.

It takes an array of languages to make the world go 'round. And we're excited about the announcement of Google's new app engine. Check it out.

By Trevor Baca, VP Software Engineering


Security best practices demand two-part identification. Credit card plus signature. Banking card plus PIN. Or, if we're in the UK, chip plus PIN. The principle is clear enough -- identity theft of a single piece of information happens too easily but theft of two loosely coupled pieces of personal information should be much more difficult.

Welcome to text messaging as one piece of the two-part security paradigm.

The people at Chase bank have been leaders in innovative uses of communications-enabled business processes (or CEBPs). Deposit more than a user-configurable number of dollars and Chase's automated prompting system can dial your cellphone and let you know. Withdraw more than a user-configurable number of dollars and Chase's automated IVR systems can dial your cellphone and leave a different message. Automated alerts are a cost effective customer experience extra with no additional overhead in call center costs.

So how are Chase working in text messaging as one part of the two-part security paradigm?

Log in to any Chase account online and click the link that says that you forgot your username or password. Nothing too exciting about username retrieval. But in Chase's new implementation, you can click a radio button to have an "activation code" sent to your phone by text message. The incoming message reads something like "Text from 242-733. Fr: Chase Online. Activation Code is XXXXXXXX. Enter is now online when prompted or in password field when logging on. Text Stop to STOP, Help for support."

It's a small detail and a great idea. We know that people look at text as the communications medium of intimacy and one-to-one contact. So it's a reasonable extension of the feelings most of us have about text to use the medium to convey small bits of personal information. And allowing "STOP" and "HELP" escape routes demonstrate extra concern for customer service in the process.

By Trevor Baca, VP Software Engineering


So much of games programming is already so virtuosic that it's frequently hard to be impressed when the next new thing rolls around. But this YouTube video on Wii head-tracking is a dramatic exception to that rule.

Aaron over in our business apps team brought the video to my attention. It features Johnny Chung Lee at Carnegie Mellon's HCI program. And it runs about 5 minutes. The part you really wanna see is starting at about 2:50 into the video where you get the first-person perspective of what IR head-tracking really allows.

Way cool. My friend John and I were talking last night about when and how game design will outgrow the conventions of cinema. That's a big question ... but it certainly seems that technical developments like the ones shown here have the possibility to help provide an answer.

By Trevor Baca, VP Software Engineering


American Public Media's Marketplace reported yesterday on work published last year in the Journal of Personality and Social Psychology by McGill researcher Mark Baldwin.

Baldwin and his team put together a special type of video game to help train people to perceive positive social situations more readily. Marketplace's description of the game has players select the one smiling face out of a larger number of total faces presented as an array. As in most games, users' skills increase with practice. And as players become better at picking out positive faces in the virtual matrix, something surprising happens -- players' levels of the stress hormone cortisol decrease. By as much as 17% in a recent study of 23 call center employees living in or around Montréal.

Who says video games are bad for you?

Baldwin's work has lead to the creation of a new startup named MindHabits, recently awarded funding to bring a commercial version of this and other social awareness games to market. Check out the website to play a demo version of the game ... though note that the site seems to be having some trouble this morning ...

By Trevor Baca, VP Software Engineering Back in Austin for SXSW Interactive 2008. Check out the booth. Towards the middle of the showroom floor. Fourth floor of the conventional center downtown.

By Trevor Baca, VP Software Engineering Jack, Brian, Ben and I got into San Diego last night... Bill may have made it too. On a late flight. ETech starts today. More info available here.

By Trevor Baca, VP Software Engineering


In parts I and II of this series we looked at couple of ways to make business go faster. Part I explored CSC (critical support conferencing) which is a Jaduka prerelease that we use to get our support team on the phone fast. And in part II we looked briefly at an install of Jaduka WARNS (weather alerts & realtime notification system) and saw how to make school and jobsite closures due to bad weather take only seconds. Both CSC and WARNS are communications-enable business processes. Both CSC and WARNS use voice to reduce human latency (the time it takes to get your team together and ready to work). Both CSC and WARNS make business go faster.

This is the third and final part of our series on enterprise collaboration. And in this post we look at Jaduka buyer verification services (BVS). BVS is the muscle behind our credit card fraud protection solution, which you can read more about here.

Fraud protection can be a pain. My mother's joked for years that all she has to do to get credit cards blocked in December is get her Christmas shopping done faster than other folks. Nothing more irking than restrictions on your credit cards that you didn't ask for and that you don't need.

But the other side is equally bad. Identity theft is a serious problem. And, despite massive attention from the press, the problem seems to be getting worse rather than better.

So what to do? How to help banks ensure the purchasing freedom of moms everywhere while offering solid customer protection at the same time?

Later this year we're working to trial a new opt-in buyer verification service with a banking partner. The service is scheduled to be available to individual cardholders on a strictly opt-in basis. Cardholders get the option to add themselves to a safety and security list maintained by the bank. Once on the list, cardholders get a call to their cellphone every time a purchase on their card exceeds a fixed dollar amount. The call prompts the card holder to enter "1" to verify the purchase or "2" to reject the purchase. A "1" signals that all is well. A "2" hooks into the bank's existing fraud control services. Unanswered calls trigger the system to try back in a configurable number of minutes.

Leveraging Jaduka BVS in this particular context makes good sense to me. We're working out integration details for the trial now and there are a surprising number of variations on the basic premise. Should the customer's PIN enter the verification callflow? How many times should the system dial for confirmation before giving up? And, of course, what are the smartest ways to integrate with the client's existing call center fraud protection resources?

But the work is looking great. And I'm a huge fan of the opt-in approach of the trial. Mom doesn't ever have to sign up for the service if she doesn't want to. But if she ever does decide she wants the added security, it's there.

And there's our third and final example of reducing human latency in a plain old business process. Fraud protection has gotten better -- and faster -- since the 1.0 Web. Automating major portions of the fraud protection workflow with Jaduka BVS should make it even faster.

By Trevor Baca, VP Software Engineering Remember snow days? Watching the 6:30 am news January mornings for school closings at the age of 12 was captivating. Plant and jobsite closings later in life are just a pain. Corporate phone trees grow large and become difficult to maintain. Delivering anything other than a "site's closed" message can be tricky -- how do you make sure everyone in the phone tree delivers the right message the right way? Message consistency is that last thing you want to worry about at 5:30 am on a bad weather day. And auditing is tricky. How do you know for sure that everyone was supposed to call actually made the right calls? Managers shouldn't have to ask these questions. Phone trees to shut down a jobsite (think construction and utilities here) are, on the one hand, the most basic thing in the world and, on the other hand, a completely unnecessary source of risk. Scripting can fix this. I've been blogging about the different Jaduka APIs recently. And the Weather Alerts & Realtime Notification System or WARNS -- which you can read more about here -- is another good example of how we've been able to communications-enable a plain old business process. How's it work? You're the decision-maker at utility company. Uou've got jobsites in New Mexico and Arizona. Bad weather rolls around and you've gotta make three site closings. You dial into the 800 number that we've set up ahead of time and the system prompts you to record your message. "Sites at ..., ... and ... are all closed for Friday the 29th of February ..." You listen and rerecord a couple of times until you've got the message exactly right. The system prompts you to press nine when you're ready to send. You press nine. WARNS dips into database tables, pulls out home telephone numbers, and dials all on-site personnel at home. "Does the system call everyone at once?" People always want to know. We do it in blocks of about 92 employees at a time. It's very fast. And there's a bit of coolness on the receiving end of WARNS, too. Employees hear their phone ring, pick up, and hear a prompt. "Incoming jobsite closure message from the ... Weather Alerts & Realtime Notification System. To accept, press 1. To reject, press 2." Employees press one and hear your message: "Sites at ..., ... and ... are all closed for Friday the 29th of February." You configure your own repeat counts and so your message probably plays two or three times. And all keypad presses -- all those 1s and 2s your employees press when they answer the phone -- report back to the WARNS administrative webpages in realtime. Which means you always know who got your messages ... and when. The name of the game with WARNS and our other CEBP solutions is reduction of human latency. Human latency is a cover term for the amount of time it takes to bring your team members up to speed and able to help out. WARNS reduces the human latency in traditional I've-called-you-now-you-call-the-next-guy phone trees enormously. The result is that everybody finds out about site closures at pretty much the same time. And you completely eliminate the wait-period where parts of the team know what's going on while other parts of the team are still in the dark. The fact that tracking reports tell you who answered the phone and who didn't is definitely useful. But the real draw is the enormous reduction in human latency. Reducing humany latency helps make sure everyone knows where they fit. And fast.

By Trevor Baca, VP Software Engineering


It's Tuesday night after the Wisconsin primaries last week. After hours. My cellphone rings. I answer and I hear a welcome chime and then a prompt. "Jaduka after-hours support. Press one to join a critical support conference. Press two to reject." I press one. And the Jaduka after-hours support system bridges me into an a conference call. The conference call started automatically just seconds ago.

I wait for a minute and the different members of our on-call support staff join the conference call one after the other. What triggered the conference call? Turns out traffic peaked on the network at exactly 8:12 pm. On a Tuesday? Tuesday night's an unusual time for a spike in traffic on any phone network. But then one of the on-call engineers speaks up. "They just announced that Wisconsin went to Obama. By double-digits."

Real-world events -- like New Year's Eve, American Idol, and, apparently, the Wisconsin primary process -- can generate lots of phone calls. If you run part of the phone network this is something you need to watch out for. A sudden spike in traffic can mean sudden strain on your network when you're talking about thousands of calls. Which is what we saw last week on Tuesday night. Traffic dropped off a minute later and we finished the conference call and went back to our evenings.

Critical Support Conferencing (or CSC) dialed us all automatically and we were able to meet in seconds. It's a neat system. It helps us manage our network. And it's fast.

So how does CSC work?

---

The answer lies in the Jaduka APIs. We have a bunch of APIs (or application programming interfaces) at Jaduka. You can check out the public API at the Devzone. And we have an even higher-octane API for our enterprise partners. And then we have dozens of other APIs that we use for all sorts of stuff internally. Our internal APIs aren't quite ready for public release. But we use them all the time. To do stuff like start conference calls.

The Jaduka Conference API -- which is internal only right now -- lets our own developers start and stop conference calls. With a single line of code. It's got room for several dozen people per call. And it's perfect for getting a bunch of people on the phone at one time automatically.

On Tuesday night our database monitoring system noticed unusually high levels of activity for that time of week and triggered one of our maintenance scripts. The maintenance script looked up cellphone numbers for all six members of the support team and then dialed all of us. Everyone pressed "1" to join and within seconds we were all working together. Working together in one "place". One very virtual place.

---

Critical support conferencing helps explain what we mean when we talk about communications-enabled business processes or CEBPs. CEBPs are just plain old business processes where it makes sense to automate communication. Getting staff on the phone to look at a potential problem is a good example. Plain old business process where we can automate communication.

CSC isn't yet a Jaduka enterprise service ... at least not yet. But interest is building. And it looks like we may do a test flight of enterprise CSC deployments soon. Stay tuned soon for more enterprise collaboration ...

By Trevor Baca, VP Software Engineering The coffeehouse and the bar. Now the club. What a good club? Bright lights and loud music. Lots of us in one place. The chance to dress up. And be seen. Tribalism? Why is the club a place we love? It's certainly not to have a conversation. Or to wrap ourselves in the conversations of others. That's what the coffeehouse is for. And it's not really to network. Or to reach out to new friends. A good club is a nerve center of fascinating people. But we don't renew old friendships while looking for the new here. That's the draw of the bar. So what else? The club is about celebration. About being with other people. Massive numbers of people. Considered as masses of other people. We engage in a type of objectification of the people around us at the club that wouldn't otherwise be acceptable. We willing turn the people around us into agents in a play or a spectacle that we come to see and wind up playing in ourselves. Brokering this fantasy seems to be an important part of celebration. Does the voice determine our experience at the club? Absolutely. By all but disappearing. When we think of the club we think finally of a place that we love that does away almost completely with the voice. The environment forbids against it. We come to listen. But to the music. And not to each other. In the club we build ourselves a place not to listen but to experience with each other. There's an element of this in the movies. Shared experience with many people around us. (And maybe more intimacy than that with the person right next to us.) But the movies and the club do different things with time. Movies package time into a box. The clubs open up time for celebration. ---Voice makes the places we love. At the coffeehouse we meet with the voice as reassurance. At the bar, as companion, helping us navigate new friendships and old. For a night out dancing we trade our voices to celebrate with others. We multiply examples and we find the voice to be a reassurance, a tool and powerful determiner of social context. These things can only help us in the experiences we seek out and the way we choose to help ourselves and others. Starting next week we'll look closely at what it means to help. And we'll ask ourselves what voice and collaboration have to do with each other.