1. Operating system hardening (applying latest software and operating system updates)
This is done first time your server is setup but we don’t update it as new patches are released. After that it is customer responsibility in case of an un-managed server.

2. Setting up free virus and anti-spam protection for your server
Virtualmin comes with sane defaults on this one, we also setup DKIM+SPF for our clients gratis for the main domain, unless you already exceeded your fairly large quota of sane & gratis support in which case it would be another Rs. 1000/- or so one time.

3. Custom firewall and intrusion detection system with automatic email alerts
Not included, firewalls are not very useful according to us, the assumption that you are running something vulnerable on any of the open ports is generally not true anymore on the Internet so the focus of exploits has shifted to open ports like http, ftp, smtp or ssh. The hostbased firewall ( iptables ) can be configured through webmin control panel or we can do it for you if required. Threat filtering on HTTP protocol by even dedicated appliances can make your website slow for end users and leave you vulnerable to DoS attacks, it is better to fix application vulnerabilities by following the best practices.

4. Performance tuning using in-memory caching available on Apache, MySQL and PHP to make sure that you get the most out of your server.
We can re-optimize the VPS configuration for Apache/MySQL etc. once or twice in a year based on your particular load/traffic characteristics. We’ll ofcourse never refuse to look into a server if a client asks us even to seek advice.

5. Free Installation and setup of various unix softwares where we spend less than 5-10 minutes of our time. Free manual reboots.

6. Optimizing your server control panel to make sure that your server is secure and functional (private DNS setup, hostnames, server contacts, log settings, etc)
We also welcome all feedback here from our clients and most likely any feature requests you make would make into the control panel eventually

Only for our managed customers

7. Server monitoring and SMS alerts using Zabbix. 24×7 monitoring of CPU, Memory, Disk I/O, Network utilization. Patches and Security Updates. Human intervention incase any of the operating parameters shoot off the charts e.g. CPU utilization
* Carte blanche from E2E management to offer memory/CPU/disk upgrades on the fly to make sure Run the Business tasks of a client are not affected for want of an upgrade approval within +33% of your CPU+Memory+Disk resources as promised in a plan, such upgrades are gratis and allow a customer not to upgrade a plan for another say 10GB diskspace or 256 MB of RAM . Upgrades are also done for our un-managed customers gratis sometimes but as a general rule we have a hands off policy with un-managed servers.

You walk into a brainjam session with your founding team thinking that you have everything crystal clear in your head, and you walk out completely fuzzy and out of focus. Worst still, every brainjam session seems to bring in angles you didn’t think of… and now you are faced with the challenge of figuring out what is it exactly that you are doing.

What’s at play here? Is this desirable? How do you balance time spent in creating the full scope of the product, vs coding the initial working prototype that gives the much needed comfort?

At e2enetworks, we are faced with this very challenge on a regular basis — whenever we have an idea or product that we need to architect and scope out, we always have to walk the fine balance between product scope and delivery.

To tackle the problem, we have laid down a few ground rules:

1. Attack the problem at the core. Jot down the rest as points that would be tackled at some point in the future — here, we spend 80% of our brain time in figuring out the core value proposition and the feature set that would best represent that. Rest, we just write out as points in our ‘vision todo list’.

2. Create an initial infrastructure as soon as possible, or use an existing one — creating a seamless workflow is one that goes a long way towards increasing productivity. While not being dogmatic about it, there are several quick things that we do:

– setup a shared space for vision, milestone, idea jamming: example: basecamp (from 37signals), google calendar/docs etc.

– setup version control and feature tracking software (subversion, git, trac, assembla) 

– create repositories for ui designs, documentation, and codebase

– mailboxes, shared wiki, and ftp space

We have noticed an enormous amount of time ‘lost in translation’ — due to lack of a workflow, the information has to be revisited, thus leading to wastage of time.

3. Build up the core objects, core db models quickly — building an initial model of the core proposition gives a massive amount of insight into the future possibilities of the system, and whether its sufficiently different from existing players in the market, or other products. In this step, going from vision, all the way to first version of database modeling has been very helpful to us. Some tools in this space:

– High level diagrams that capture the workflow, the product overview / sitemap, the object relationships
– Omnigraffle, Mindmap, Freemind, and anything else that can help visualize object relationships… even Powerpoint can help (since its easy to draw boxes and arrows).
– Azurri Clay DB Modeling Plugin for Eclipse
– Model interrelationships in Rails, testing them using script/console

4. Create parallel processes for prototyping, feature jamming, product design. While the team operates in an agile fashion, it helps when everyone has a key responsibility — it their personal job then to make sure that it happens.

5. Create a staging environment asap — going from prototype dev to stage is key to discovering bugs / nuances in the system that would otherwise remain confined in the development (edge trunk) machines. The ideal workflow that has worked for us is — create a stage release every evening (nightly build), and let that be the system that gets tested next day while developers create the next version.

6. Play with at least one production deployment version — you discover a lot of holes in the system early on, if production deployment is tested as soon as there is some sort of working prototype. While this was inhibitive in the past due to costs involved, that’s no longer the case if one chooses cloud deployment where you pay as you go.

– it helps in figuring out number of frontend, monitoring, database servers that might be needed
– it helps in understanding areas that might need optimization later (although we are heavily against premature optimization).
– it helps discover any quirks the deployment scenario might bring into the picture.

The hosts we find interesting: Amazon EC3/S2, Slicehost, Google Appengine, Railsmachine…

7. Testing. This is an obvious one, so I am not going to go deep into it — but the gist of it is, with RAD frameworks like Rails, testing should be rigorously built into the development process, so that at the end of first iteration of the product, there are test cases already prepared. 

Keeping the above ground rules in mind for any particular product, idea or vision has helped us in minimizing time lost in random iterations that lead to nowhere… or from getting lost in the forest of ideas that inevitably emerges from a core one.

However, most important of all, one should never lose sight of the whiteboard. No amount of milestones and to do lists can capture the synergy of quick discussions that are chalked out on the whiteboard, and then eventually make their way into system specs.

The idea is simple: I create a set of microprograms (I fondly call them ‘brats’!), each of which have their own selfish agendas, decision processes, survival rules, and their own I/O probes that constantly monitor the surroundings for resources they need (and perish soon if they don’t find them)… and then I spawn each variety of agent multiple times and see an overall behavior emerge.

The first set of such agents are already sweating their necks, making highly stupid decisions… and yet surviving, but my goal is to now take it a step further.

I now I want to go in two directions:

- train a GA or reinforcement learning based system to figure out the initial configurations.

- spread the agents across multiple computers on the web, thus creating a virtual cluster, an ecosystem that would be very interesting to study.

You may ask, why is this any different than Cellular Automata, the system that Stephen Wolfram brought back into fad through his book ‘New Kind of Science’? The fact is, though similarities may exist on the surface, a deeper inspection would reveal that my agents are far more inspired by ant colony / multi-agent based emergent behavior systems than CA — the only difference being, I am trying to far more flexible about the toolkit, instead of sticking with just pheromones!. Also, at the end of the day, I am just trying to learn my tools!

[Tools used: Ruby, RubyGems, MySQL, a lot of caffeine, and box dvd set of The Wire]

To complicate things further for them, let me make a wild prediction: Facebook will be the first real social web search backed with a semantic engine, that reaches widescale adoption. 

I will not make any further comments about why I think so — but by the second quarter of next year, the Google would have to seriously rethink on ways to protect its pie. Period.

1. Smart indexing: every single wikipedia entry contains enormous amounts of information that can be linked / referenced to other sources on the web. For instance, for an entry on Berlin, a service that truly taps the potential, would be able to instantly look up highly contextual information from across the web on ‘Berlin’ (with the understanding that the word refers to a city, and nothing else). This is possible today, but only in a very limited fashion. This is exactly where the ‘semantic search’ battle is unfolding at the moment.
2. Lightweight data wrappers: spread across the web of data would be a web of data-wrappers: algorithms that understand and intelligently mine the relevant data. Interestingly, in my preliminary experiments, perl-based technologies stand out: with the vast cpan repository base, and super fast regex, it provides the exact framework one needs to rapid protoype a text-mining agent.
3. Sentence parsing: Why is it important? The power to automatically parse through blobs of english text, and generate semantic-web uris is going to be sweet. But then, lets put it in writing once and for all — natural language parsing is and has always been hard. The emergence of localized versions of the english language isn’t helping either. However, for sources of information that are well-formatted, the challenge is slightly easier. Take a look at the montylingua project here, along with the other heavyweights in this field such as wordnet. To quote: MontyLingua is ‘a Free, Commonsense-Enriched Natural Language Understander for English’. It actually works very well in most circumstances. In fact, I am planning to write a full review of the technology right after this post.
4. The DB and the efficiency question: SPARQL, one of the most popular query formats for the semantic web, is very close to SQL in its structure. Yet, the semantic web is all about triples: subject predicate object. Since relational DB is all about key-value pairs, this leads to inherent inefficiencies (read, joins or other hacks using caches) which could have been avoided if the db world had evolved slightly differently (interestingly, Google App Engine’s data store is very powerful here — try it out if you don’t believe me).
5. Revenues, revenues, revenues: of course, the big daddy of all… duh?! Why would someone pay to have shitload of relevant information? Think..?!! What if you could offer the user products which are exactly what he might have been looking for? What if you could understand what he has been searching for all along. Semantic Web is not just a technological gibberish from the business standpoint, it actually has some tremendous revenue potential built into it. Forget Google Adwords and Google Adsense, they just spam you on the right side of the screen. You can do much much better.

I wish to list out a few more, and then write a very structured post on this. However, that would have to wait a bit.