Do we need a JSON data type?

I recently read a few articles raving about how good PostgreSQL is.  One article in particular explained how great it is that they have a JSON data type.  I wondered exactly what that would mean for developers, and whether Uniface needs one too.

The PostgreSQL documentation states that JSON data can be stored just fine in a text data type, but that a specific data type for JSON adds specific validation for JSON strings.  The documentation then adds that there are related support functions available.  Indeed there are JSON operators and functions that massage data between JSON strings and table rows and columns.  Suppose that you have a use case to exploit these functions, should you use them?  The simple answer for a Uniface developer is “of course not”.

Looking at those JSON support functions I would suggest that you can write Uniface functions / local proc modules to manipulate and transform data in similar ways.  Uniface Structs and the new 9.6.04 structToJson and jsonToStruct statements are particularly helpful for this.  So, provided that there is no extreme performance advantage in doing such manipulation on a DB server, it would not be a good idea to tie your application to a specific DB vendor, and lose that DBMS independence that Uniface gives you.  Bear in mind that there is no JSON data type in the current SQL Standard from 2011, and the major RDBMS vendors have not found a need to add such a non-standard extension.

Since we do have JSON manipulation tools, there is another consideration, based on our experiences with XML.  How do we validate the meaning of data transported by JSON?  With the xmlstream data type (and supporting proc statements) we have DTDs.  With our Structs transformations we have XML schema validation support.  With Uniface entities, we have the full support of the application model.

What is missing is a JSON Schema mechanism.  Thus I would suggest that if there is no supporting validation mechanism, there is no point in having a specific data type for JSON.

That situation may change in the future.  There are Internet Engineering Task Force (IETF) drafts available for a JSON Schema standard.  If you want to anticipate this as future standard, you can use this online tool to generate a JSON Schema:  http://www.jsonschema.net  from a sample JSON data stream.

At this time, to use this draft JSON Schema, you will need to write a validation module yourself.  However, you may be able to validate the data based on the Uniface Application Model.  After loading the Struct with the jsonToStruct statement, you may want to prepare the Struct for using the structToComponent statement.  Since 9.6.05+X501 the structToComponent supports a /firetriggers command option, which causes the Pre Save Occurrence and Post Save Occurrence triggers to be fired, thus allowing you to do further occurrence based validation or manipulation.  Of course the entities that you use for this purpose can be dummy entities created for this purpose, modelled or not.  This would avoid the need to reconnect with the database.

Hopefully we now have enough tools to deal with JSON data, without the need for a new data type.


Part 2: The threat of the Start-up and how traditional development teams can look to fight back

By Clive Howard, Principal Analyst and Bola Rotibi, Research Director, Creative Intellect Consulting

Part 2 (read part 1 here)

Appreciate the skills, knowledge and assets that you have

Once an organisation, however large, adopts a culture in which the development and IT teams believe that things can be done quickly but still within high standards of quality and compliance, then they can compete against their smaller challengers. This cultural shift can be difficult with often resistance coming from those entrenched in the old ways. Some may fear that the new processes will make them redundant. This is why organisations have to tailor new processes to their strengths and considerations such as governance have to be taken into account.

For example, when moving to Agile it is important not to be too fanatical about a certain methodology such as Scrum. The best Agile environments are those where the approach is tweaked to suit the organisation’s skills, needs and concerns. A start-up does not have to worry about large legacy investments with years of domain knowledge built around them. An enterprise most likely will and so that knowledge (people) needs to be retained. Equally some projects may still require a more Waterfall style approach due to the nature and scale of the systems involved. Enterprises therefore need new processes that embody Agile execution practices, but they must be sensible and balanced in their application.

Don’t forget operations

Agile will help developers add new features more quickly but it is only part of the overall process. Moving to CI and CD processes will create a development and operations environment that allows reliable and stable software to be released quickly. Embracing the concept of DevOps (the removal of artificial barriers between operations and development teams and finding a new working relationship that benefits the entire software process) will reduce the friction between the development and operations teams and so help to get new releases into production more quickly.

In addition the development teams need to make sure that speed does not sacrifice quality. Something that start-ups have learned is the importance of testing. The growth in popularity of Unit Testing and Test Driven Development (TDD) has been fueled by this. Enterprises need to make sure that they have the necessary testing tools, capabilities and culture in place – something that has been lagging within enterprise development teams. By making testing a constant within the development process they can increase the quality of code. Often in traditional Waterfall environments the test phase was squeezed and so in reality quality and software stability, was sacrificed.

All that glitters is not gold

Finally there is the question of technology. Start-ups have become synonymous with new technologies such as PHP, Ruby on Rails, Django and a host of other platforms, frameworks and services. They tend to gravitate towards these as they believe that they allow them to work more quickly and so focus more time on concerns such as the User Experience of the product. In reality some of these are immature and result in more time being spent firefighting than working on making the product better. Enterprises often deal in legacy software and far larger usage requirements than many start-ups have to deal with initially. A MySQL database may work great with a certain amount of data but as Facebook discovered at scale it can pose challenges. So, don’t throw out the Oracle or the IBM Database just yet.

That does not mean that technology is not an issue in the enterprise. With applications now needing to be deployed to an ever increasing number of platforms and devices the underlying technology choices will impact speed of delivery. Having a solution that places as much logic into a single codebase utilising a common language, skillset and tools will have great time and cost saving benefits. As many organisations are constantly discovering, having to maintain multiple codebases in different languages and tools that effectively do the same thing is increasingly time and cost intensive. Therefore approaches such as hybrid mobile development or model driven development will reap rewards especially over time.


Random Number Generator in Uniface

If you are not aware of it, Rik Lewis has a great blog about Uniface. I was quite interested in a recent post about the generation of random numbers. We don’t *officially* have this in Uniface. I don’t know why we don’t (probably something related to C++ across multiple platforms from the dark ages).

I’m interested to find out if this is something that we should put in the product (I’m assured it’s not too difficult), but I’m curious to know some use cases. The obvious one would be to generate primary keys, although there are other ways to do this.

But also things like how long would it need to be? 10, 20, 50 characters or should it be something like $random(10) which specifies 10 characters?

Comments welcome..

The threat of the Start-up and how traditional development teams can look to fight back


By Clive Howard, Principal Analyst and Bola Rotibi, Research Director, Creative Intellect Consulting

Part 1 in a 3 part series

Increasingly enterprises are feeling threatened by far smaller companies referred to as technology “Start-ups”. These micro organisations are often innovative and agile with a focus on user experience to quickly engage and win users. It is important to state that not all of these companies are the same. An academic “start-up” in the UK university town of Cambridge, could be very different from a two person company in Shoreditch, a borough in London that is being hailed as the hub of the UK Silicon Valley.

Start-up appeal

However it is most often the Shoreditch style of start-up that large organisations most fear.

Typically these companies are small, pure play and driven by acquiring as much market share as possible in as short a time as possible. To do this they have to focus on what attracts users more than issues that enterprises typically concentrate on like governance. If these companies get it right then they can rapidly become incredibly successful with large market valuations.

The poster child for “start-ups” is of course Facebook but there are many others, some less well known, that have seriously disrupted industries and have become worth billions of dollars in the process. In their wake often lie the demise of well-known “bricks and mortar” companies such as Blockbuster (film rental provider), HMV (music, film and games) and Jessops (photo processing and photography equipment). As these companies fell others looked on and began to think that they could be next.

That begs the question as to how any organisation either weighed down by decades of building software in a certain way (often very slowly) and without the environment to make change quickly  can respond to such innovative and nimble competition? How each company answers this question will probably decide whether they are still around 5 to 10 years from now.

It’s the culture, stupid

The first issue that enterprises should look to address is process. Many enterprises still follow software development processes created for a different era when software ran on desktops and quarterly upgrades were considered regular. Most start-ups can get a product to market in weeks (sometimes days) not years and upgrades come every few weeks, daily or faster. These small organisations have often adopted Agile methodologies and concepts such as DevOps, Continuous Integration (CI) and Continuous Delivery (CD) (see CIC report on Continuous Delivery and DevOps : link – http://www.creativeintellectuk.com/research-library/#id81). Such processes inherently allow for software to be altered and delivered very quickly. The dynamics of the workflows truly embody the notion of a software factory with repeatable and predictable deliverable outcomes.

These new processes are not beyond the enterprise and certainly Agile is rapidly gaining ground. The traditional Waterfall based approach that large organisations have pursued for years is being replaced. This is by no means easy and requires transformation programs that involve departmental restructuring, new roles (such as Scrum Masters) and training. Most significantly it requires a change in culture as well as a mind shift that is willing to address and shake up the underlying politics. It is here that start-ups truly have the advantage. Often populated with young developers who usually have to take on many different roles (including operations) they have a culture of delivering quickly.

Once you have teams that can turnaround new features and push them out to production rapidly then you have an environment in which innovation can thrive. Innovation can be difficult when an idea has to go through multiple tiers of management and takes months to design, develop and deliver. Instead, using faster processes and idea can be pushed out, tried and then refined based on user feedback very quickly. In a world of ever increasing platforms and different devices, the ability to create an app for a new platform or form factor very quickly allows that business to be more creative.

Uniface web workshops, closing the generation gap

In March and April this year, a new initiative–the Uniface Plusprogram ran in the Amsterdam Uniface Lab. The Plusprogram originated from the Benelux user group Face to Face to deliver Uniface / Web workshops for Uniface developers. A special so-called PlusMembership allows Face to Face members to participate in the program which consisted of two parts:

  • HTML5 CSS3 and Javascript concepts
  • Uniface web components like HTML5 widget and Uniface Dynamic Server pages

The workshops were given 8 weeks in a row on Tuesday and Thursday evenings from 5:30 PM unitil 9:00 PM including a “light” Pizza or Sandwich dinner. Participants could choose to go on Tuesday or Thursday as the content for each evening was the same.

Tons blog

Five evenings were all about Web basics and the latest and greatest in HTML5/CSS3 and Javascript technology. These workshops were done by a young Master in Computer Science (Tim–who also happens to be my son) and has a lot of both hands-on as well as theoretical web knowledge. During those evenings it was very nice to watch a 24 year old teaching a class room full of very experienced Uniface developers Modern Deployment using Git.  Looking at the evaluation we held afterwards, the students very much appreciated and learned a lot from these sessions. Every evening consisted of 90 minutes theory, 60 minutes “hands-on” and a quiz. Having a quiz at the end of each session is quite common for Web trainings these days, but for most of the students is was pretty new. I think this a little “generation gap” between a “raised with gaming and Google generation” and a more traditional “schoolbook” generation.

Tons blog 2

The final Web Workshop evening explained modern Javascript libraries like Node.js and NoSQL databases like Mongodb. The other 3 evenings were given by Uniface consultants and developers and explained the concepts of the new Uniface HTML5 widget, DSPs and responsive Web programming which is a “must have” for deployment of Mobile Websites. These workshops showed that Uniface already has many possibilities to use the Web technologies which were showed in the first part of the program. The PlusProgram was very successful. During the 8 weeks over 90% of the participants showed up and we also got requests from other countries to organize the same program locally. For now we are almost ready to organize the same program in Belgium after the summer holidays.  For other countries we will determine on a case by case basis if/when/how the program will be presented. If you have any questions about the Plusprogram please feel free to contact me!