<![CDATA[ZET UNIVERSE - BLOG]]>Thu, 01 Aug 2024 11:18:03 -0700Weebly<![CDATA[Zet Universe and Next Steps]]>Sat, 15 Sep 2018 15:56:58 GMThttp://www.zetuniverse.com/blog/zet-universe-and-next-stepsSeveral years ago I've got a privilege to give a TEDx talk, on Functional Illiteracy, in Moscow, Russia:
This talk was one of my opportunities to re-think what's the next step for Zet Universe, and to get me back to the roots.

Back To The Roots

My original passion for the industry was that in it, you can create what everyone else would have call magic. Isn't it magical that with a simple phone in late XIX century you could reach out to Europe from US? Yeah. Same goes for video calling today, when workers from Central Asia make calls via Viber from Moscow to their relatives left there, in their homelands.

Another thing that excited me about IT industry was the idea of augmenting us with technology. Like Archimedes once said: ​
Give me a place to stand, and I shall move the world.
Archimedes
The Archimedes's lever in the modern world is technology, and if you are in the industry, you have a chance to move the world as you please.

So, yeah, what I essentially wanted was to move the world. Why?

Since early days of my life, I've seen my grand pa building things. I've seen Russia being slowly disassembled to the pieces, and getting them sold, and then I've seen those who remade our country, and made parts of it (however small they could be) better.

It's so easy to destroy, and so hard to build. 

And those who build, those who leave clean emptiness with a meaning, are equal to Antique gods, in my imagination.

With that, and with technology, I want to see us, humans, to become gods. 

Gods capable of building entire worlds, capable of building anything we want.

Experiments: Building Better Worlds within Games

In childhood, I played Civ 2 and SimCity. Later - C&C, Transport Tycoon. Recently I've played Anno 2270, Surviving Mars, Aven Colony, Frostpunk, Fallout 4 (settlements mode), and other things. 

All of these games are sophisticated simulators of our world - or, at least, of some of the aspects of it, and of different scales.

In Fallout 4 you build settlements on a Boston Greater Area, in Frostpunk you work with a small city and a large territory around it. In C&C games you build a base on a large map. In SimCity you deal with entire city. In Transport Tycoon you deal with a greater area, where you not only build transportation systems, but also deal with making cities, too. In Surviving Mars and in Aven Colony you deal with colonies on the planet's surface. In Anno 2270 (one of the most economical simulators I've ever seen) you work at the scale of entire planet and its satellite, Moon. Finally, in Civilization, you deal with the planet, but scale of your operations is thousands of years (if you are lucky).

In all of these games, you get a chance to become a god. 

With wisdom (or cheats) you can grow your own worlds, with the given tools. You can enjoy the prosperity of your cities, share photos of your miniature settlements, cities, transportation systems, and other things.

But what you can't do is making changes to the real world. 

All of these games provide you with an immersion, however sophisticated, but an immersion. 

When you are out, the world around you haven't changed. 

There are still problems around the world. 

So if you didn't fix your house's roof, you'd still loose warmth of your home. And if you didn't buy food, your family would starve. 

So, to become a god, a real god, playing games won't help. The worlds you'll create there will be virtual.

You need to build real technology-based tools that would augment you into a god to build worlds.

With these thoughts, I've turned my head to IT industry.

From Microsoft and Google to Zet Universe

Microsoft and its founder, Bill Gates, captured my attention with Bill's dream of having a WinFS - an object-relational storage. Microsoft also used the idea of empowering people's potential that I've fallen in love with:
Your Potential. Our Passion.
Unlike Google of mid 2000-s, Microsoft communicated it's belief that computing should be local (on your PC), and that you could do so much with your computer.

Wow, so a big company wants to give everyone tools to realize their potentials? I'm interested. 

It believes these tools should be with them, be local, be private? I'm hooked.

To me, this idea of using a PC as a technology lever to augment us, humans, looked like a perfect reason to join Microsoft.

After joining it, I've asked a few folks from Windows, Office, and MCS to help me in advocating the importance of building such lever on top of the Microsoft client and cloud platforms.

We called this idea a "Windows Semantic Platform", and we wrote a ThinkWeek paper for BillG and other senior execs to read it. Yeah, this paper outlined a vision for a system that would span across both client and cloud, but I believed that client had to be rich, and that cloud would be a pipeline to connect those clients together when needed. I believed that both sides would mirror each other.

Long story short, after spending almost 4 years at Microsoft, organizing a SIG on Context-aware Computing, doing a seminar with Gordon Bell (who authored the book on Total Recall, or your digital memory) giving a closing talk, I've left the company, and spent about a year at Google.

Google seemed to be a place where people really wanted to build this digital memory of the world, and of every single person. But what I've found was that despite of this dream, Google solely wanted to make their system work only online, without any place left for client computing.

With my understanding of importance of keeping personal data personal, I couldn't stay. 

Also there, at Google, I've understood that I shall build this lever with my own hands, and thus the desire of starting Zet Universe was born.

Zet Universe

To build things, you need to understand a set of simple ideas called project management. Without it, without a plan in your head, you most probably won't build anything really big.

And I wanted to create tools to aid my users and me in building big things.

So, a project management aspect was essential. 

Second aspect was learning. You can go just that far with your passion. When provided with the necessary knowledge, you can do practically everything (as "Mysterious Island" by Jules Verne taught us).

Zet Universe, as I look at it today, was an opportunity to build a tool that would augment a human in learning and managing projects.

In my vision of the future, every human would have a wearable computer, always there, that would: 

* record everything its user sees and hears, and provide a digital memory to its user,
* be a place to organize user's thoughts and ideas in a structured form of the knowledge graph,
* be a passive tool that would aid user in her everyday activities, 
* be a system that would do activities on user's behalf, 
* be proactive and help user when the need for help arises, 
* protect user from the unnecessary interruptions,
* be a system that would encourage user to grow in her area of choice.

All in all, I wanted to have a technical lever that would augment each human in the world, and give him or her a chance to achieve more.

And one of the most important things here, I strongly believed that such a tool should be private, should be always with its user, and not be hosted in the cloud.

I've shipped two versions of Zet Universe, going from a rough prototype (~4GB, hard to download, hard to install, hard to get up and running), to a small (20mb) installation, running easily on x86 machines.

Zet Universe became an amazing tool for organizing user's thoughts and ideas about different problem spaces.

In Zet Universe, you can now create endless project spaces, add entities from Wikipedia (using Dandelion API), add your files and folders, add web pages, and create your own entities using either built-in kinds, or after expanding it with your own ones.

With these entities, you can define connections between them (either manually or by asking Dandelion API to extract them automatically from the documents and web pages added to the project space), and capture all of the knowledge graph you've created with the graph entities within the system.

In essence, Zet Universe became a tool to build your own knowledge graph, with an ability to connect it to the bigger knowledge graph based on WikiData/Wikipedia.

Till this very day, Zet Universe is a local application that runs entirely on your PC, and with the necessary plugins, it can extract information locally (but in order to connect them with Wikipedia/WikiData, it'd require some sophisticated work), without any need to be connected to the Internet.

Afterall, your personal knowledge graph should remain private.

After spending 5.5 years on Zet Universe, however, I thought it was a good time to take a pause, and do some stuff in the industry. And so I did.

Yandex

A lot had happened since that TEDx talk, and especially in the last year. Still being driven by the ideas of building AI-driven tools (see DARPA CALO and PAL projects), I've been invited to join Yandex's Mail team. And so I did.

It was a fantastic time at Yandex.

I've been busily building a continuous semantic indexing pipeline for Yandex.Mail, and working on a ToDos and Reminders scenario for Yandex's voice assistant, Alice.
​​
While the pipeline became a backbone for making such scenarios as reminding you about your flights and changes in them, ToDos and Reminders scenario became one of the foundational ones for Yandex's Alice. This project involved ~20 teams (Alice is hosted in several Yandex's mobile and desktop apps, and ToDos and Reminders involve 1 frontend and two backends; not mentioning all of the other stuff), required a lot of cooperation across those teams, and aligning them in order to ship this scenario to the public.

During this work on ToDos and Reminders, I've realized how standardized voice assistants are these days. They share a lot with Zet Universe, in that they have a pipeline for working with user requests, they extract information (slots) from those requests, they do the requested operations (when possible). 

In many ways, digital assistants are a perfect component of the earlier vision of that "human augmentation system" we've described in the Windows Semantic Platform ThinkWeek paper, and what was shown in the following it Productivity Future Vision (which I've referenced a few times in this blog before).

What's Next?

As I look back at what was accomplished with Zet Universe and Yandex's Alice, it's clear that integrating Zet Universe and a voice assistant to work on the same machine is the next big step for me.

You can do a lot of stuff with tools like IFTTT and digital assistants.

My requirement for such assistant is to make sure it can work offline, on-a-chip, without any need to access Internet.

Surprisingly, there is a such assistant.
Despite of the fact that Alice, Alexa, Cortana, Siri, and Google Assistant all need Internet connection to function, a brave Snips.AI team from Paris, France, just made it!

It's called Snips.

Read more about their technology here. E.g., Snips' take on NLU, their deep dive into ASR on the embedded system (also look at their paper), and so on. This is all a fascinating stuff, and I'm super excited about the work this team is doing.

One little thing I find especially cool is their approach to data generation. At Yandex, we used internal tools based on language rules to generate training texts automatically, but configuration was done by the engineer.

Snips offers the same functionality as a paid service where you can pick the number of samples, the way they are produced, and then, suddenly, you've dramatically increased the quality of your intents! 

As of me, I've spent a work week trying to get their tech to work.

By the end of it, I made it.

Snips platform works perfectly on my DIY voice assistant hardware kit:
If you want to learn more about setting it all up, read my step-by-step instructions here.

As of the other developments... 

​Stay tuned!

P.S. I'm in no way affiliated with Snips.AI.
P.P.S. Snips team is hiring!
]]><![CDATA[ANNOUNCING CHANGES TO INSIDER PREVIEW PROGRAM]]>Wed, 08 Feb 2017 17:21:11 GMThttp://www.zetuniverse.com/blog/announcing-changes-to-insider-preview-programIt was almost two years ago when we've started the Zet Universe Insider Preview program. We've shipped more than 20 builds in the first year, between July 2015 and May 2016. We've received a lot of amazing feedback, thanks to you, and we've been encouraged to keep working on the product, all thanks to you, our Insiders!

​Our goal for the milestone we've started working on last Fall is to bring a reliable, high-quality version of Zet Universe for you. As you already have noticed, we've stopped shipping new builds since mid-2016. This time, however, we will disable the associated web services that support Insider Preview builds of Zet Universe.

THE OTHER 90% OF SOFTWARE DEVELOPMENT EFFORT

As our mentor, Skip Walter (Chief Product Officer @ FTI Consulting, Inc.) likes to say, there's a difference between the prototype and the actual product.

"It's one of the fundamental challenges in new product, balancing the drive for ever more useful functionality for customers with the need to have a very high quality product." Indeed, thanks to all the feedback we've got over the last year and half, it's now time for the next 90% of the software development effort - RAAMPUSS:

RAAMPUSS is an acronym that is a short hand for eight aspects of a quality product:
  • Reliability
  • Availability
  • Adminstratability
  • Maintainability
  • Performance
  • Usability
  • Scalability
  • Security

This is the set of our high-level requirements for the next version of Zet Universe, internally called v3, and we are putting all of our energy into this milestone. This also means we'll have to disable the associated web services that support Insider Preview bits.

USING ZET UNIVERSE INSIDER PREVIEW AFTER 2/25

What will keep working
The good news are, you'll still be able to run your builds of Insider Preview after February 25, but only after changing your settings file like this:
  1. Exit Zet Universe Insider Preview (right click on its icon in system notifications area on taskbar and choose Exit)
  2. Go to %appdata%\Zet\
  3. Find user.v2.settings file
  4. Open it in Notepad (or any other plain text editor)
  5. Find line AppMode=0, and 
  6. Change it to 1 (AppMode=1)
  7. Save file
  8. Run Zet Universe Insider Preview
Congratulations. After following these steps, you'll be able to keep using your builds of Zet Universe Insider Preview after February 25, 2017.

What will stop working
Several things:
  1. Updates - you will no longer be able to download the updates to Zet Universe Insider Preview after February 25, 2017
  2. Download the setup - you will no longer be able to download setup.exe of Zet Universe Insider Preview after February 25, 2017
  3. Receive "unread" notifications from Dropbox plugin for files from Dropbox in your Zet Universe Insider Preview project after February 25, 2017​
We want to say thank you for your support in this journey, and we are looking forward to see you with us once we'll ship the new bits of Zet Universe v3 later this year.

​Stay tuned!
]]><![CDATA[ZET UNIVERSE DEMOS - 2009-2014]]>Sat, 10 Dec 2016 20:52:45 GMThttp://www.zetuniverse.com/blog/zet-universe-demos-2009-2014Zet Universe originally was envisioned by our founder, Daniel Kornev, during the development of the Productivity Future Vision 2019 video made by Office Labs Envisioning Lab between 2007 and 2009.

First prototype was created in several May nights in 2009. It took 2.5 years for our founder to make a decision to leave corporate world and establish Zet Universe, Inc., as the company, to productize those original ideas, and since that time we've shipped several versions of the product. As we are moving forward with the new iteration of the product, we find it invaluable to look back at its history.

Today we are excited to share several product demos, never shared before, created between 2009 and late 2013.

2009 | ORIGINAL PROTOTYPE

2012 | ALPHA VERSION

2013 | PRE-BETA VERSION

Special thanks to Alexander Popov for his narration in this video.
]]><![CDATA[FIGHTING FOR UI PERFORMANCE - 2012-2013]]>Fri, 09 Dec 2016 20:58:56 GMThttp://www.zetuniverse.com/blog/fighting-for-ui-performance-2012-2013When we started this journey almost 5 years ago, it wasn't clear to us just how long it could be. Building a product like Zet Universe means fighting with lots of challenges, and one of them is UI performance.

While we are hard at work making new milestone of Zet Universe faster than ever, we would like to share our progress in increasing performance of navigating in the infinite zoomable canvas, as it is seen in these two videos, made by the end of the first and by the end of the second year of this journey.

2012, DECEMBER - ALPHA

2013, DECEMBER - PRE-BETA

]]><![CDATA[SPATIAL UX: SOFTWARE & HARDWARE DECADE LATER]]>Wed, 02 Nov 2016 12:15:54 GMThttp://www.zetuniverse.com/blog/spatial-desk-for-spatial-user-interface-decade-laterUpdate (2/20/2022): This blog post was updated in February 2022 to refresh links to two original Productivity Future Vision videos that were removed from the web by Microsoft sometimes in 2020-2022, as well as to share a screenshot of a Spatial Desk in reality built by our Founder in early 2021.
Today you know Zet Universe not only as a product, but also as a long journey towards creation of a true spatial user environment, designed to make work with large amounts of information intuitive and simple.

Originally, Zet Universe was inspired by our founder's previous work at Microsoft (2007-2010), made in collaboration with the Office Labs and Office Labs Envisioning teams.

Between 2007 and 2009, those teams worked on many aspects of the future of Productivity, including but not limited to new device form factors (be that new phones, tablets, walls, or big desktops), new kinds of team software (precursors to Slack and Yammer, spatial environments, etc.), and so on.

Some of those ideas were released by Office Labs via the (now defunct) website officelabs.com, and experiments like "Canvas for OneNote" became truly popular among the Office power users. But, certainly, the biggest visible outcome of that work was the Productivity Future Vision video itself, shown below.

SPATIAL UX: CANVAS, PLEX, ETC.

In 2007-2009 that idea of the spatial environment caught a lot of attention across the company; ideas of Jef Raskin and his ZUI were singing in the future visionaries' hearts, as well as the hard work of the folks behind first Microsoft Surface (table).

That amount of attention led to creation of several projects, with most of them concentrated in the Office Labs organization.

The spatial environment itself, internally also called as "Plex", or "Canvas", was designed to be a zoomable infinite canvas, one that enables users to intuitively organize their information into projects ("spaces"), and each of those spaces to have rich information objects placed directly on canvas. By zooming in and out, one could jump from small details to the "bigger" picture, following their thought process, and by organizing objects spatially, one could organize them by their meaning to each other.

These simple principles have been forever imprinted in the teams' hearts and minds, and thus many variations of this spatial environments were born in the process:
  • Canvas for OneNote (made by Office Labs)
  • Windows Canvas/Project Universe (made by our founder, Daniel Kornev)
  • pptPlex (made by Office Labs)
  • Plex (made by Office Labs Envisioning)
  • Code Canvas (made by MSR)
  • Debugging Canvas (made by MSR)
  • (some other internal undisclosed prototypes)

Windows Canvas, made by our founder, later became a foundation for Zet Universe, while other apps either got their app momentum (like Canvas for OneNote), or at least got their attention in the press:
Picture
Windows Canvas/Project Universe (2009)
Picture
Canvas for OneNote (2009)
Picture
pptPlex (2008)
Picture
Plex (on TouchWall) (2009)
Picture
Russ Burtner showcasing Plex (on TouchWall) (2009)

SPATIAL DESK

Alan Key, a computer science pioneer, once said a very important thing about designing software:
People who are really serious about software should make their own hardware.
And so, among many things Office Labs and its partners worked on in the 2007-2009 time frame, several hardware prototypes were born (designed by John Snavely & Russ Burtner).

It was not just a set of devices, but a dream of a new set of devices for productivity, for information workers: new tablet, new phone, big wall, and a Spatial Desk.
Picture
Spatial Desk, as shown in the Productivity Future Vision (2009)
Picture
Spatial Desk (3D render, 2008)
The key idea behind the Spatial Desk was its ability to provide its user with a giant touchable surface, that could be usable both in the horizontal and vertical form, making it easy to work with the large spatial environment that other teams were hard working on.

By 2010, the Envisioning team got first hardware prototype of the Spatial Desk:
Picture
Daniel Kornev (our founder), toying with the Spatial Desk prototype (2010)
Picture
Spatial Desk hardware prototype, Microsoft Envisioning Center (2010)
Picture
Spatial Desk hardware prototype, Microsoft Envisioning Center (2010)
Picture
Spatial Desk hardware prototype, Microsoft Envisioning Center (2010)
As the first Productivity Future Vision has been released to the public in 2009, Envisioning Team continued its work, refining its concepts, and, of course, Spatial Desk has been upgraded.
Picture
Spatial Desk v2, as shown in the Productivity Future Vision (2011)
Picture
A photo of Spatial Desk and its user during the filming of the Productivity Future Vision (2011)
In 2011 a second video, a follow-up if you like, was made by the Envisioning team, continuing this approach:
However, a shift in direction has happened, and Microsoft moved to a simpler set of devices, starting with Surface Pro, a 2-in-1 tablet. 

WHERE ARE WE DECADE AFTER?

Today, almost a decade later, we are not there yet, both in terms of software and hardware.

MICROSOFT: DECADE LATER

SOFTWARE

Internal Microsoft efforts ended up with the Zoom feature in PowerPoint, available for the Office 365 customers, while the rest of the projects vained.
Picture
Zoom for PowerPoint (2016)
Another Productivity Future Vision video has been produced recently, illustrating new categories of Surface devices with it, including but not limited to 2-in-1 tablets, PixelSense wall, and Surface Studio:

HARDWARE

That amazing Spatial Desk, shown above, haven't made it, yet. The closest thing Microsoft got to that Spatial Desk is Surface Studio, shown in the latest Productivity Future Vision.
Picture
Update (2/20/2022): Our founder built a real prototype of the spatial desk using the market-available hardware in February 2021:
Picture
This setup is enabled by one 27" multi-touch screen with pen input (Dell Canvas 27", bought from Amazon), and two 15.6" multi-touch screens (w/ VESA mounts) bought from AliExpress.

ZET UNIVERSE: DECADE LATER

In the meanwhile, we at Zet Universe have went through 8 iterations, and are currently working at the 9th iteration, making Zet Universe simple and small again.

ITERATIONS #1-#8 (2009-2016)

Picture
Iteration #1 (2009)
Picture
Iteration #2 (2010)
Picture
Iteration #3 (2011)
Picture
Iteration #4 (2012)
Picture
Iteration #5 (2013)
Picture
Iteration #6 (2014)
Picture
Iteration #7 (2015)
Picture
Iteration #8 (2016)

ITERATION #9 (2016-2017)

Picture
Standalone app
Picture
Windows Shell integration
This was a very long journey (8 years since first prototype, 10 since first ideas, 5 since company creation), and it's a kind of journey not everybody could make.

However, it's great to see that some of the hardware envisioned for the Information Workers of the future, is now available.

As of us... We've made a lot of iterations, learned a lot in the process, tried many different things... We've made mistakes. We've got some positive feedback.

Today, we have a fantastic team, and we are looking forward to impress you with our newest software, delivering you a spatial environment you could only dream about!

Stay tuned!

ITERATION #10 (2017-2020)

Update (2/20/2022): After joining Yandex, our Founder made an experimental integration of Zet Universe v2 (2013-2016) with Yandex Tracker.

After leaving Yandex, our Founder made more experimental integrations with Snips (now acquired by Sonos), ABBYY Information Extractor, Integrum, and other Russian and international information processing/cognitive/Conversational AI systems. The last integration was made for a Conversational AI startup our Founder was consulting in a CPO role in late 2019-early 2020.

ITERATION #11 (2020-NOW)

Update (2/20/2022): ​After joining DeepPavlov in a CPO role, our founder focused his attention on setting up and building a product organization there. However, as his role got more and more incoming information ready for processing, a concept of Zet Universe became viable once again. Current experiments are focused on building some sort of a "Canvas for Slack".
]]><![CDATA[SCALING UP EFFORT]]>Mon, 15 Feb 2016 21:36:16 GMThttp://www.zetuniverse.com/blog/scaling-up-effortFrom the beginning, Zet Universe was envisioned as a knowledge management and analytic platform, that could be used to solve different problems, from organizing personal information, and learning new things in the spare time, to researching different topics, analyzing competitors, tracking work projects, and more.

One of the biggest values Zet Universe possesses is its ability to work in an offline mode, and work locally. It's different compared to other famous analytic platforms, such as Palantir Gotham, or Quid, and others, as these platforms typically require you to have a constant network connection and (except Palantir Gotham) require you to have a high speed connection to their cloud computing services.

Zet Universe, in contrast, allows you to have your projects data with you, residing on your computer, as you are on the go, riding on a car to your client, or getting to a new destination by train or a plane. Zet Universe contains a powerful semantic infrastructure that is capable of extracting and analyzing data from your own projects while working from your own machine.

In order to make this possible, Zet Universe requires some good hardware to work, ideally with no less than 4GB of RAM, Intel's Core processor, and SSD drive. But good hardware is only part of the equitation, as we also have to make Zet Universe's software tailored to its specific tasks.

As you, our Insiders, work with Zet Universe, we learn more and more about our current product design's strengths and weaknesses, and as you tell us more and more about your needs, we get better understanding of how we should improve the product to make it being more relevant for you.

This blog post is a story of our Scaling Up Effort, and in this series we will discuss two areas of this effort, Storage Layer and thumbnails cache.

SCALING UP STORAGE LAYER

In 2013, Air Gesher and Danielle Kramer of Palantir made a talk at the Strata Conference 2013, named "AtlasDB: ACID Transactions for Your Favorite Key-value Store". In this talk they discussed a new bolt-on layer for a key-value stores (distributed or otherwise), a system in use at Palantir called AtlasDB, which sits on top of either LevelDB (on local computers) or Cassandra (on distributed systems).

Palantir Gotham's interactive analytic core, which was originally built with a traditional RDBMS as its backing store, was hitting the limits of economical and easy scaling. They needed to move a distributed backing store for scalability, and they need to also run Palantir Gotham on local machines as well. To solve this problem, Palantir development team designed and built a special transactional layer running on top of the key-value stores, enabling Palantir Gotham to work with way, way larger data collections.

Meanwhile, at Zet Universe, we were hitting a similar obstacle. The v2 of Zet Universe (the one we've been developing since mid-2013) was originally built with a file-based storage as its backing store. Each time you add a new document or ask Zet Universe to track a folder, special items called entities are created, and each new change is stored in an individual JSON file. This approach was dictated by the goal to make Zet Universe data being easily syncable between user's and his/her colleagues PCs using fast and convenient synchronized cloud storage systems like Dropbox, OneDrive, or Box.

After running a series of experiments with early customers, we've got to a conclusion that we should employ a custom synchronization solution. In late 2014 we've run another series of experiments in an effort to scale up our storage layer (including RDF-based stores, SQLite-based, generic Key-Value stores, and others), and it became clear to us that most of these solutions are not really relevant to our problem, and we decided to keep using the existing one, to learn more about the product's real world usage to guide us in building a more efficient backing store.

Thanks for a 6 months of the Insider Preview Program, and for your invaluable feedback, we are now glad to share the good news: starting with the February 2016 build, Zet Universe will be now using a more efficient backing store.

DESIGNING NEW STORAGE LAYER: UNDERSTANDING DATA AND DEVELOPMENT AT ZET UNIVERSE

We take a research approach to building software. Instead of just using the existing solutions, we look at the wide variety of different systems running in production, recognize their patterns, compare them to the patterns of Zet Universe, and ask, "What technical solution would be the most efficient for our user in his or her work?" To answer this question, we, very much like are counterparts at Palantir, use a holistic understanding of how low-level data integration, scalable data stores, API layers, and an entire suite of user interface tools, when properly integrated, create an efficient and simple user experience. For us, it is very important to put the technology out of the way from our users, and yet make each piece of it to work at maximum level.

Certainly, when we have components that already exist to serve our needs, we are glad to use them - be that Lucene as a high-quality open source search engine, or WPF as our presentation framework. But when we identify a capability gap, we build new things.

DESIGN GOALS

As it was said in the beginning of this blog post, one of the core goals for us is to make sure you can keep using Zet Universe not only when you have a high-speed internet connection, but also when you are offline or are occasionally connected to the network. There are, however, several other goals we've set for us as extremely important to fulfill:
  • It should work in the offline mode, keeping current user's projects fully accessible, and newly added local data being processed within the system while working offline,
  • It should be kept small (currently is installation is 11MB worth),
  • It shouldn't not require admin rights to install and use,
  • It should be able to work on 32-bit machines (and it has to run as a 32-bit application due to use of specific libraries),
  • It should allow working with relatively large data sets (from 10K up to 1M of objects within the project).

UNDERSTANDING OUR DATA MODEL

Zet Universe's data model is, in many ways, unique, as it has aspects of both spatial and temporal databases:
  • Each object in its data model has its place on the 2-dimensional information space, and each object can have a history of changes.
  • At each moment of time, Zet Universe is showing you only a current snapshot of its data.
If you have say 10K objects, Zet Universe stores approximately 100K historical change records about those objects, but it needs to load only 10K objects into memory.

ROUND 1: EXPLORING SQL DATABASE SERVERS

Originally (in Alpha version, 2012-2013), Zet Universe used a complex approach, with data stored in the Microsoft's SQL Server 2012, and it was accessible via a locally deployed Web Service.

This approach had several benefits:
  • Both user's data and metadata are stored within one data store
  • Database is running in the separate process, and only required chunks of data are sent to a client
  • SQL Server provided us with a built-in full-text indexing, semantic similarity search, proven backup solutions, and more.

After first deployments within the team and first customer deployments it became clear to us that this approach isn't really efficient; Alpha version deployments also helped us to formulate some of the design goals:
  • It should be kept small (currently is installation is 11MB worth),
  • It shouldn't not require admin rights to install and use,

ROUND 2: JSON FILES

Once we've formulated the new design goals, we've decided to find the most simple solution for data store, and, combined with the idea of making metadata and project data synchronizeable via consumer- and business-grade cloud storage platforms like Dropbox, OneDrive, and Box, we've ended up with a simple approach:
  • User can have multiple projects,
  • Each project is technically a folder, and it can have one or more objects (entities) inside it,
  • Each entity is represented as a collection of individual storage records, saved as JSON files
This approach worked quite well for some time:
  • Zet Universe loaded file names on each startup,
  • Zet Universe than picked only those it needed to get a current snapshot of the spatio-temporal database (those files make altogether)
  • Each new record means a new separate file on a disk.
However, this solution wasn't scalable, as we wanted to move from hundreds of objects within the single projects to tens and hundreds of thousands of objects. We've also found that Microsoft's antivirus, Windows Defender, was running checks on our files each time Zet Universe was launching, which means that each application's start up could become slower due to extensive antimalware checks being run by Windows Defender.

​We've continued the search.

ROUND 3: EXPLORING EMBEDDED SQL AND NOSQL STORES

In late 2014 we've decided to return to SQL-based solutions, employing SQLite as a possible embedded database. 

Unfortunately, this approach wasn't really right for us: SQLite runs a query processor, maintains tables, indexes, and other artifacts of the relational databases, which were irrelevant for our data model. 
Between late 2014 and early 2016 we've tried other embedded solutions, like Google's LevelDB, Symas Lightning Memory-Mapped Database, ESENT (Microsoft's ISAM database), Brightstar, and other solutions.

In general, we've got to the following conclusion:
  • Each embedded solution loads at least part of the database in memory (think Google's LevelDB), which isn't acceptable for us,
  • Most of the embedded solutions have a lot of functionality we didn't really need (think full-text search, SQL Language support, triple stores, and other things).
In the long run, even the most-efficient key-value store would become a bottle neck to us, as it would use the precious memory for its data structures, and, most possibly, would provide us with functionality we won't need.

ROUND 4: EXPLORING FILE-BASED SOLUTIONS

In late 2014 we've researched an opportunity to persist the entire data graph to the disk, but decided against it. As it was mentioned earlier in this post, we don't need to load everything into memory; all we need is to load the current snapshot (where current is based on the current date and time which changes every moment as we speak).

We've returned to the file-based solutions. This time, we wanted to use one file on 

One of the possible ways would be using a ZIP or TAR archive, and keep metadata inside it. However, the internals of ZIP archive logic made it clear that updating its often would lead to large memory usage which was undesirable for us. 

Another option was to use virtual file systems. Windows has a built-in support for VHD disks since as early as Windows 7; however, to create new VHD disk or to attach an existing one, user needs to have admin rights, which goes against our design goals.

After running a technical discussion with our advisors, we've got a recommendation to look at the old Windows functionality, "Locking and Unlocking Byte Ranges in Files", which is also used in the old OLE/COM Structured Storage format, in it's implementation called compound files. This is the same format Microsoft used for old Microsoft Office files.

The internals of this format are simple: essentially, it's a sort of a virtual file system (FAT). It has root, which can contain storages (think directories) and streams (think files). It has an internal hierarchical index, and in order to read specific parts of the file you need to specify an internal path to it, instead of loading the entire file into memory.

This sounds pretty much like our situation. We could store our metadata right within the compound file, and we could save historical records into it, and we could stop worrying that the entire file could have a size much larger than the available physical memory (a bottleneck for the rest of the embedded stores).
Indeed, this is the solution we've ended up with for now. Each project's metadata (objects, properties, relationships) is now stored in its own compound file, and as each such file's size is limited only by the file system limits, this approach is quite scalable for our needs.

WHAT'S NEXT?

Our current approach is rather straight-forward. We migrate old JSON files into the compound file, and then we form a spatio-temporal index during each application's start. This was relevant when we used a file system, as it was expected that new files could appear while the program wasn't running. Now that the synchronization will be working within the application's boundaries, we will cache the spatio-temporal index within the compound file as well to speed up the application's startup.

Another area for improvement is to store the entire object graph of the current snapshot, and load it on the startup, and then use the spatio-temporal index for new items, to update the snapshot.

These improvements won't be available as part of the February 2016 build but will follow with the upcoming builds.

SCALING UP THUMBNAILS CACHE

Historically, most of the data tracked by our users in Zet Universe was in the form of folders and files. A visual metaphor of an infinite zoomable space made showing meaningful miniatures of those files a critically important piece of the overall frictionless user experience we wanted to provide our customers with.

With this goal in mind, we've employed an aggressive thumbnail caching strategy. However, although this strategy worked when users needed to work with files and folders only, it became the bottleneck for us in our effort to support much larger data sets.

Since the February 2016 build, Zet Universe now maintains a very small thumbnail cache, storing only commonly used thumbnails (generic thumbnails for each kind of data), and loads thumbnails for the rest of the items from disk in an asynchronous pattern. 

This helped us to support, for example, 10 projects with more than 16K of records entirely in-memory, and spending nearly 200MB of memory for the entire application. Given that there are some visuals that define the user interface itself, this is a pretty good use of memory; previously it was easy to hit 1GB of memory with a way smaller amount of projects and records.

ZET UNIVERSE TODAY

Zet Universe has an efficient and scalable local data store based on the compound files, where the only limitation is the file system limitations. It also has a new entity thumbnail system that caches only commonly used thumbnails, and loads the rest of thumbnails in an asynchronous pattern.

Stay tuned for part two of the Scaling Up Effort series, where we'll do a deep dive into the next steps of further improving our local data store.
]]><![CDATA[WELCOMING FLIGHT 20 OF ZET UNIVERSE]]>Mon, 15 Feb 2016 21:35:44 GMThttp://www.zetuniverse.com/blog/welcoming-flight-20-of-zet-universeHi everyone,

We're rolling out Zet Universe Insider Preview Build 5889.2386 right now to Zet Universe Insiders. As it was said in the previous blog post, since shipping the second (and last) January 2016 build of Zet Universe, we've changed our focus from the Dynamic Ontologies to our storage layer in our effort to scale up Zet Universe. And so, this new February build has a historical significance for us, as it is a first fruit of this effort.
Getting Started:
  • If you haven't signed up for the Zet Universe Insider Program, start here.
  • If you are already have Zet Universe, the build is available now. If your build is older than 5786.2836, grab the full installer.  Otherwise, you can go to Settings --> Update, and click on "Check Now" button to get it now.
  • Total download might range from a few hundred kilobytes up to 11MB, depending on the number of updates you've applied already to your copy of the Zet Universe Insider Preview.
  • Once the build is downloaded, Zet Universe will begin the installation. Once it will finish, it will ask you to restart the app.
  • After restarting the app, Zet Universe will show you a "Migration Wizard", asking you for permission to migrate to a new data format. Click "Begin migration", or read this blog post before doing a migration.

WHAT'S NEW IN THIS BUILD?

The most important things you need to know about the February 2016 build are:
  • It contains a more scalable and efficient data store,
    • To continue working with Zet Universe, you need to migrate your existing data to this new format, for which a new built-in migration wizard is provided in this build,
  • It uses significantly less amounts of memory,
  • It contains a bug fix for connecting Dropbox account.

If you are a new user of Zet Universe, you can just skip reading the rest of this blog post, and install the latest version of Zet Universe right away.

Otherwise, if you are an old user, please read the next part of this blog post. If you want to learn more about the reasons for changing our data store, check our our post on the subject, "Scaling Up Effort".

MIGRATING TO A NEW STORAGE LAYER

MIGRATION WIZARD

After installing the February 2016 build, you will see the following wizard dialog:
Picture
At this point, you'll be invited to either begin the migration right now, or postpone it until next launch.

POSTPONING MIGRATION

If you are in hurry to see your projects, or you don't have time for migration, you can easily postpone the migration. In this case Zet Universe will load your projects as usual, and it will have a new ending in its window's title: "Legacy Storage Mode":
Picture
Next time you'll launch Zet Universe, you'll get a Migration Wizard dialog again.

MIGRATION

If you are ready for migration, click on the "Begin Migration" button in the Migration Wizard dialog. In our test environment (11 projects, 17K storage records; Core i5, SSD) migration took about 5 minutes, but in general results might vary.

During migration, the dialog will show you the progress:
Picture
Once migration will end, you'll be notified with a message "Migration Completed" and you'll see a button "Restart":
Picture
Click on "Restart" button and let Zet Universe complete the migration effort. It will automatically close and relaunch again. After that you will see your projects in the same state you left them before the migration. Congratulations!

CURRENT LIMITATIONS

As this is the first public release with the new data store, we encourage you to always use the notification area icon to properly shutdown Zet Universe, as it saves data during the shutdown.

We will bring back more incremental data saves in the upcoming builds.

WHAT'S FIXED?

You can again add Dropbox account to Zet Universe - starting with this build, you will be again able to add Dropbox account to your Zet Universe. Due to internal changes on the Dropbox side, the previous approach we used to connect it to Zet Universe no longer worked, and we've made the necessary changes on both sides to restore this functionality.

WHAT'S NEXT?

In the coming builds we will bring back incremental data saves in our new backing store, continue upgrading our plugin platform, and return to our work on the Dynamic Ontologies (including custom tags), the related Rules Engine, and return to our Product Roadmap. Of the other things, we will also continue working on the SDK.

Thank you so much for being with us, for your continued support and participation!

As always please continue to give us feedback, suggestions, and problem reports via the UserVoice forum, use the Zet Universe Insiders Group on Facebook to connect with other Insiders for help and tips on problems you hit.

Thanks,
Daniel and the team
]]><![CDATA[ANNOUNCING ZET UNIVERSE INSIDER PREVIEW BUILD 5866.39662]]>Tue, 26 Jan 2016 14:43:29 GMThttp://www.zetuniverse.com/blog/announcing-zet-universe-insider-preview-build-586639662Hi all,

We are coming very fast to the end of January, and it means there's only one more build to share with you this month, and this build is available to you starting today.

Unlike the previous build which had only a change in the search syntax (albeit an important one!), in this build you'll have a unique chance to take a look at the behind-the-scenes of your projects information space - the ontology.
Getting Started:
  • If you haven't signed up for the Zet Universe Insider Program, start here.
  • If you are already have Zet Universe, the build is available now. If your build is older than 5786.2836, grab the full installer.  Otherwise, you can go to Settings --> Update, and click on "Check Now" button to get it now.
  • Total download might range from a few hundred kilobytes up to 11MB, depending on the number of updates you've applied already to your copy of the Zet Universe Insider Preview.
  • Once the build is downloaded, Zet Universe will begin the installation. Once it will finish, it will ask you to restart the app.

WHAT'S NEW?

Some of you might have noticed that if you would go to Settings-->Options and choose "Show Views pane demo user interface", you'll see the new pane in the left sidebar, called "Views". The currently selected view is called "Zoomable Space", and there are hints that more views, like "Project Hearbeat" and "Mind Map" will be also available. However, this is just a mockup right now, as these views are not yet a part of the everyday builds.
Picture
Well, starting with this build, we are glad to introduce a second working view to Zet Universe, and, surprise, it won't be any of these two additional ones, as shown in the demo UI. Instead, this new view, called "Ontology", will enable you to look at the behind-the-scenes ontology of your projects. You can either click on the "Ontology" button in the "Views" area of the sidebar, or...
Picture
Or, you could open "Properties" of any of the items in the project space, and click on the entity's Kind to see its place in the ontology of the system:
Ok, now that you know how to get into this new view, let me give a short introduction into what it is all about.

ONTOLOGY AND KINDS

One of the founding principles of Zet Universe was and is the idea that it is an environment in which one can combine data from different places together, for a variety of purposes. A typical example could be use of this environment to gather intel on a particular subject.

Imagine having a need to analyze your potential competitors in a particular market; then, in Zet Universe you can collect various things you've discovered during your research: links to the useful webpages, PDF files, PowerPoint presentations, pictures, and other things. No matter what kind of data you have, you can see it all in one place, and you can organize it visually, no matter of its origin.
Picture
In this screenshot, you can see an example of how such a project space could look like for someone collecting an intel on intelligent personal assistants in general, and Microsoft's Cortana specifically.

Some of the items here are captured web pages, others are PDF files, others are videos and images. You can also see several Sticky Notes that capture key thoughts of the user on the subject, placed near the items of interest. User can quickly find any of the items here by using her visual memory, but also by using a full-text search as Zet Universe indexes everything that is added into it.

In order for Zet Universe to understand all of these different items, it has to understand what makes them different from each other. One of the ways to categorize them is to use file formats (e.g., jpg, png, bmp - images, mp4, avi - videos, mp3, wma - audios, etc.), but sometimes file formats are irrelevant, and sometimes we want to have more specific items, like "people", and "organizations". This means we need to build some sort of a data model that brings a categorization of data into "kinds" as we call them, that brings also a structure (e.g., "photo" is kind of "media", and "media" is kind of "kind"), and brings other important things such as properties (e.g., each "kind" has "title"), and relationships (e.g., a "document" kind can have an "author"). In general, such data models are called ontologies.
Ontology, by a definition, is a formal naming and definitions of the types, properties, and interrelations of the entities that really or fundamentally exist for a particular domain of discourse.
Generally, it is expected that any ontology is to be designed in such a way that features of its model closely resemble the real world.

There are many applications of ontologies in the information science; for instance, ontologies are one of the foundational elements of the Semantic Web, and ontologies are also an amazing way to capture and formalize our knowledge of the particular problem space or domain. In many ways, ontology could be also called a vocabulary of sorts, and, indeed, a well-designed ontology resembles an encyclopedia.

Speaking of the encyclopedia, I recall the fictitious Chinese taxonomy of animals described by the writer Jorge Luis Borges in his 1942 essay "The Analytical Language of John Wilkins". 
Wilkins, a 17th-century philosopher, had proposed a universal language based on a classification system that would encode a description of the thing a word describes into the word itself -- for example, Zi identifies the genus beasts; Zit denotes the "difference" rapacious beasts of the dog kind; and finally Zita specifies dog.
In response to John Wilkins's proposal, Borges described an example of an alternate and rather funny taxonomy, supposedly taken from an ancient Chinese encyclopedia entitled "Celestial Emporium of Benevolent Knowledge":
​
The list divides all animals into one of 14 categories:
  • Those that belong to the emperor
  • Embalmed ones
  • Those that are trained
  • Suckling pigs
  • Mermaids (or Sirens)
  • Fabulous ones
  • Stray dogs
  • Those that are included in this classification
  • Those that tremble as if they were mad
  • Innumerable ones
  • Those drawn with a very fine camel hair brush
  • Et cetera
  • Those that have just broken the flower vase
  • Those that, at a distance, resemble flies
As it was said in of the beautiful works on the ontology of the everyday things, though, this fictitious “encyclopedia” captures the potential hubris and arbitrary nature of many of our concepts of “what there is.” The encyclopedia is amusing and idiosyncratic but not very useful except as a psychological profile of the apocryphal compiler of the work.

However, having a well-defined vocabulary is a fundamental step in solving our goal of enabling Zet Universe to combine data from the different data sources, and it's a great opportunity for us to explain our approaching to building such a vocabulary in Zet Universe.

ZET UNIVERSE'S DYNAMIC ONTOLOGY

Zet Universe's vocabulary starts with the definition of a Kind. Previously, in the Alpha timeframe, we called it a Thing, but as our goal is to make our ontology connected to the RDF and standard ontology language, OWL, we decided to rename this element to Kind, and make it a subclass of the standard owl:thing. Now, what is the Kind thing all about?

We live surrounded with things. These things can be physical ones coming from our environment, or they can be products of our imagination, or, finally, be digital objects created as elements of virtual world built by computer applications, websites and games. Things we deal with, starting from documents and other files in our folders, to users in Facebook and emails in our inbox, seem to be natural to us. We constantly make different interactions with these things, switching from physical world to a virtual one back-n-force, referencing to them in our endless collaboration with friends and work colleagues. The patterns of interacting with digital objects were brought from the physical world by Internet and computer pioneers, and now these patterns are highly interconnected as we constantly transfer our experiences between our real and digital lives.

Instead of a computer strict understanding of classes and instances, Kinds in Zet Universe are more familiar to the users; and as in the Metro Design Language, kinds are content.

THE RATIONALE BEHIND THE KINDS

In general, file formats are the way files are distinguished from each other, but not every single piece of information is stored as a file, and we as people don't usually think in terms of file formats. The ontology of Zet Universe, as it was said above, is based on the things we interact with on the daily basis, and thus its kinds represent the categories of these things.

By bringing the kinds into Zet Universe, we explain it how to deal with different kinds of things; a kind definition explains Zet Universe how to show it (visually) in the Zoomable space; it explains how to name this kind (e.g., a "Photo"); it explains how to open items of this kind; it explains how this kind is related to the others.

KINDS IN ALPHA AND IN INSIDER PREVIEW

In the Alpha timeframe, we had only three kinds:
  • Topics,
  • Files,
  • Web Pages. 

In the Insider Preview timeframe (which is running right now), we have a significantly expanded list of kinds, including but not limited to:
  • Agent (Organization | Person | People Group | User)
  • Calendar Entry (Appointment | Meeting)
  • Communication (Call | Email | Instant Message)
  • File Folder
  • Generic File
  • List
  • Media (Document | Music | Picture | Video | Web Page)
  • Project
  • Visual Cluster

​These are the built-in kinds of our vocabulary. They are, in many ways, a continuation of our thinking circa the Alpha Timeframe (Topic is now a Visual Cluster, File is either a Generic File or a Media, etc.), and they were influenced by the kinds defined by Windows operating system (and mapped to the same file extensions as in Windows). This means that if you got used to search for "kind:photo" in Windows Explorer to look for pictures, the same query in Zet Universe will yield the same results.

However, although we've been inspired by the Windows Shell ontology, our ontology is not a try to explain and define everything, and it's not carved in stone.

DYNAMIC ONTOLOGY

As it was stated above, our goal is to make it possible to integrate many different kinds of data from many different sources into a coherent whole that reflects how people naturally conceive of information.

To make it possible to reflect different people's ways of thinking about the information, we decided to make a compromise: although there is a default vocabulary of Kinds, both developers and users will be able to define and re-define the ontology on the fly.

Part of this functionality is already available in the product. For instance, if you use a Dropbox app within Zet Universe, your ontology will have a "Dropbox Folder" kind which is a child of a "File Folder" kind, and if you use Sticky Notes, your ontology will have a "Sticky Note" kind as a child of a "Media" kind, and so on. Both our own and third-party developers will be able to extend the default ontology with the new kinds.

Third-party Kinds have the same role of the building blocks of Zet Universe as the built-in ones. For example, Dropbox and alike folders can participate in the project space as external data sources, while Sticky Notes are items created within the project spaces, and so on.

In the future builds, you will be able to define your own ontologies so that Zet Universe could be better suited to your specific needs.

ONTOLOGY VIEW

In order to see the ontology of Zet Universe, you can use our brand new "Ontology" view:
Picture
In this view, there are two tabs, illustrating the different aspects of the ontology browser:
  • Items, and
  • Properties.

ITEMS BROWSER

Picture

By using the Items tab, you can not only the hierarchy of the default Zet Universe's ontology, but you can also see actual items that represent the selected kind:

PROPERTIES BROWSER

By using the Properties tab, you can see the properties relevant to a selected kind. 
Picture

CURRENT LIMITATIONS

It's important that as this is the first release with the Ontology View, there are many limitations. For instance, you can't see custom properties relevant to a specific kind. There is no way to browse relationships. There is no way to define new kinds, or new properties, or relationships. These features will be available in the upcoming builds and will be announced separately.

WHAT'S FIXED?

File Path is Too Long Exception leads to a fatal crash - starting with this build, you shall no longer experience this fatal crash. We've significantly improved checks for the very long file paths, so that your work experience with Zet Universe would not be degraded by such annoying errors.

SOME OF THE KNOWN BUGS

  • ​Initial Office document upload to the OneDrive folder won't be picked up while Zet Universe is running. Zet Universe will pick up Office files uploaded to OneDrive after restart. This bug is being investigated, but there's no ETA at this point. We are sorry for this bug.

WHAT'S NEXT?

In the coming builds we will start working on the more scalable storage subsystem for Zet Universe, we will upgrade our plugin platform with the latest improvements from our Architect, Danila Korablin, and we will return to our work on the Dynamic Ontologies and the related Rules Engine, as we've defined that in our Product Roadmap. Of the other things, we will also continue working on the SDK, and we will start reaching out to the selected developers among Insiders to ask them to evaluate the SDK.

Thank you so much for being with us, for your continued support and participation!

As always please continue to give us feedback, suggestions, and problem reports via the UserVoice forum, use the Zet Universe Insiders Group on Facebook to connect with other Insiders for help and tips on problems you hit.

Thanks,
Daniel and the team
]]><![CDATA[JANUARY BUILD NOW AVAILABLE TO THE ZET UNIVERSE INSIDERS PROGRAM]]>Wed, 13 Jan 2016 12:17:20 GMThttp://www.zetuniverse.com/blog/january-build-now-available-to-the-zet-universe-insiders-programUPDATE - grab an updated January 2016 build, 5857.11090, to restore Dropbox functionality in Zet Universe. See details below.

Hi everyone,

First of all, I'm very glad to wish you a Happy New Year! It's exactly 5 months now since we've started the program (July 13, 2015), and it was an amazing ride with you, Insiders. As people like to say, we are just getting started! As you might have noticed, we've been keeping silence since the very end of November 2015, and for a good reason. Our top priority goal was to restructure our underlying plugin platform, extract it into a separate package, and move all of our currently available plugins to our new platform.

Well, we've done that! We've also made a few improvements in this first build of 2016, build 5857.11090, though, and I can't wait to tell you about them.
Now, if you’re unfamiliar with the Zet Universe Insider Program, this is our community who is helping us build Zet Universe. If you’re not a Zet Universe Insider yet, we’d love to have you join – see below. Also make sure you read the list of known issues at the end of this post before getting started.
Getting Started:
  • If you haven't signed up for the Zet Universe Insider Program, start here.
  • If you are already have Zet Universe, the build is available now. To get it, you can go to Settings --> Update, and click on "Check Now" button to get it now.
  • Total download might range from a few hundred kilobytes up to 10MB, depending on the number of updates you've applied already to your copy of the Zet Universe Insider Preview.
  • Once the build is downloaded, Zet Universe will begin the installation. Once it will finish, it will ask you to restart the app.

​Notes to existing users:
Update your January Build to avoid issues with Dropbox
If you are running on the original January 2016 build, 5856.5513, you may encounter an error with Dropbox: it's folder isn't shown when trying to add a Dropbox folder to your project; Zet Universe crashes when a new file was added to a Dropbox folder watched by your project(s); Zet Universe crashes after re-connecting to the Internet. Please update to the 5857.11090 to stop encountering these bugs.

Use full installer to update if your build is older than 5786.2836
If you are running on a build before 5786.2836, and you haven't been getting updates for a while, the "Update Now" button might be unavailable to you. Don't worry - you can simply install Zet Universe again by downloading its installer from the Set Up Your PC webpage. This won't break anything, but will give you the latest release available. If, however, you are running a 5786.2836 build or later, you won't be affected by this issue.

WHAT'S NEW WITH THE SDK AND NEW PLATFORM?

Zet Universe has a modular and extensible architecture, and in order to make it's architecture available for the ISVs, we've moved out the SDK library from the rest of the product, and unified plugin platform by integrating the SAL.Flatbed, our proven and tested plugin infrastructure.
Picture
Please note that the conceptual architecture you can see above is the first draft (representing the current state of things), and you can expect us to continue updating the product, and expanding its services and functionality, as it was outlined in The Roadmap blog post.

INTRODUCING ZET UNIVERSE SDK

Starting with the January 2016 build, your Zet Universe will be running using our brand new SDK (ZU.Core.Developer.dll) and plugin platform (SAL.Flatbed.dll).
Picture
As ISV developers, you will be able to introduce your own Apps (to connect custom data sources with Zet Universe) and your own Semantic Processors (to extract and classify metadata for rich interaction and further analysis).

​In the coming months we will publish our SDK library and first code samples to the GitHub, and will make it available for the first Developer Insiders.

WHAT'S NEW IN THE PRODUCT?

Now that you've got a chance to take a look into the internals of Zet Universe, let's discuss the new feature we've introduced in this build: Search Filters​.

Technically, these filters have been introduced in the middle of September, 2015, as it was described in the blog post. However, we've heard from you that you wanted to make more strict searches, and we're glad to deliver the updates to the search experience.

FILTER AND KEYWORD MATCHING

Search for all items and filter them by filter "note" kind:
kind:note
Picture
Search for term "cortana" and filter by "note" kind:
kind:note cortana
Picture
Search for terms "cortana" and "deep" and filter by "note" kind:
kind:note cortana deep
Picture
Search for items that have term "deep" and might have term "cortana", and filter results by "note" kind:
kind:note cortana +deep
Picture
Search for items that have both terms - "deep" and "cortana", and filter results by "note" kind:
kind:note +cortana +deep
Picture

FILTERS AND WILDCARD MATCHING

Search for all items that have words starting with "Cort" and filter them by "document" kind:
kind:document cort*
Picture

THIS IS JUST THE BEGINNING

By implementing Search Filters, we wanted to re-use the Best Practices widely used by Google and Microsoft in the search experiences of their products like Bing, Google, Windows Explorer, Gmail, and others.

We are still working on the Advanced Search Syntax for Zet Universe, and we would be glad to hear your thoughts on it.

WHAT'S FIXED?

Last Modified date is now shown in the Search Results and is used as the sort by field in the "By Date" search results view.
Center Screen button works again.

PDF Document Thumbnail Extraction works again after changes introduced in the last year's build.

After loosing connection with the Internet, and then reconnecting to Internet, Zet Universe will crash with a message "Could not load file or assembly 'System.Net.Http.Primitives, Version=1.5.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified." This bug has been fixed as part of the updated January 2016 build, 5857.11090. Please update your Zet Universe to 5857.11090 to get this bug fix.

​Dropbox folder isn't shown when trying to add a Dropbox folder to a project - This bug has been fixed as part of the updated January 2016 build, 5857.11090. Please update your Zet Universe to 5857.11090 to get this bug fix.

Zet Universe crashes upon a new file is added to a Dropbox folder that is part of one of the projects in Zet Universe - This bug has been fixed as part of the updated January 2016 build, 5857.11090. Please update your Zet Universe to 5857.11090 to get this bug fix.

Zet Universe identifies and includes partial Chrome and Edge files (while they are being downloaded) into the project files - This bug has been fixed as part of the updated January 2016 build, 5857.11090. Please update your Zet Universe to 5857.11090 to get this bug fix.

SOME OF THE KNOWN BUGS

  • ​Initial Office document upload to the OneDrive folder won't be picked up while Zet Universe is running. Zet Universe will pick up Office files uploaded to OneDrive after restart. This bug is being investigated, but there's no ETA at this point. We are sorry for this bug.

WHAT'S NEXT?

In the coming builds we will upgrade our plugin platform with the latest improvements, and we will return to the Product Roadmap, working step by step on our Vision. We will also continue working on the SDK, and we will start reaching out to the selected developers among Insiders to ask them to evaluate the SDK.

Thank you so much for being with us, for your continued support and participation!

As always please continue to give us feedback, suggestions, and problem reports via the UserVoice forum, use the Zet Universe Insiders Group on Facebook to connect with other Insiders for help and tips on problems you hit.

Thanks,
Daniel and the team
]]><![CDATA[ANNOUNCING THE LAST FALL'15 BUILD OF ZET UNIVERSE]]>Mon, 30 Nov 2015 18:59:48 GMThttp://www.zetuniverse.com/blog/announcing-the-last-fall15-build-of-zet-universeHi everyone,

Today we are releasing Zet Universe Preview Build 5813.5130 to Zet Universe Insiders. Like the last few ones, this build is really focused on bug fixes and general improvements; at the same time we've made one colossal change to the product that we feel being really proud of, and we can't wait to share this change with you. What is that change? Oh, we've changed the entire way of... Wait. First things first.
Getting Started:
  • If you are new to the Zet Universe Insider Program and haven't installed Zet Universe, yet, please visit the Set Up Your PC webpage to download the latest build's installer.
  • Otherwise, as usual, the newest build is available via the magic of Zet Universe Update system, powered by Squirrel for Windows. To get it, you can go to Settings --> Update, and click on "Check Now" button to get it now.
  • IMPORTANT: If you are running on a build before 5786.2836, and you haven't been getting updates for a while, the "Update Now" button might be unavailable to you. Don't worry - you can simply install Zet Universe again by downloading its installer from the Set Up Your PC webpage. This won't break anything, but will give you the latest release available. If, however, you are running a 5786.2836 build or later, you won't be affected by this issue.
  • Total download might range from a few hundred kilobytes up to 10MB, depending on the number of updates you've applied already to your copy of the Zet Universe Insider Preview.
  • Once the build is downloaded, Zet Universe will begin the installation. Once it will finish, it will ask you to restart the app.

WHAT'S CHANGED?

As you might recall from our previous blog post, by making it possible to import large folders, we've brought a few new bugs. One of them was this:
  • Zet Universe will be unresponsive after importing a large number of items (think a folder with 500+ files).
  • ​We are aware of this issue, and we are actively working on fixing this issue.
So, as we said above, what's changed is a big thing. BIG thing. No, really. It's THAT BIG!

While working on this bug, we've learned that we should change the way how Zet Universe treats changes to your data.

​And by doing that, we've also significantly improved UI responsiveness during heavy data processing.

Wait, what? Why did this bug's fix require such a fundamental change?

Ok, calm down. Let me give you an example.

Imagine you are working on a project with your colleagues. You have set up a shared Dropbox folder, and you keep work documents inside it. Now, you've imported this folder into Zet Universe, so that you could track changes to your documents in it, in a visual way. So far so good. Next time someone will edit your file, Zet Universe will mark it as unread, and mark this time as the Last Modified time. Sounds good, right?

Well, almost. But there's a catch. Once you will open this file, or simply move it in the space, or once Zet Universe extracts full-text data from this file, Zet Universe will mark this file as updated, again. But this time, file will remain the same! Why is then it's marked as updated?
Technically speaking, the reason is simple.

Every time you ask Zet Universe to track your project folders, it creates a metadata (data-about-data) entity for each of your files, and when you move the file in the project space, or do any kind of change to your file, its corresponding entity is changed, and this time of change is used as the Last Modified Time.

This is extremely helpful as it makes it possible for Zet Universe to track a history of all operations applied to these entities, but it's, actually, an internal information that you might not really need to know about in the first place.

What's more, Zet Universe is doing a lot of data processing in the background (extracting thumbnails, full-text, adding it to the full-text index, extracting keyphrases, etc.), and prior to this build, we've been updating the last modified value for each entity each time any of these operations happened. As the modern computer is pretty capable of fast data processing, Zet Universe's Navigation Pane had to re-sort all entities after each change, which led to UI lags at best, and fatal crashes in the worst cases.
Obviously, your files shall be marked as updated under certain conditions. Think when you edited a file, or one of your colleagues did, or if the file was renamed. But all other operations (moving in the space, adding to or removing from visual clusters, indexing, etc.) are not, strictly speaking, about your file. They are about what is being done with its metadata, not the file itself.
​
And so, starting with this build, we mark your files as created OR modified only if:
  • File created
  • File contents changed 
  • File renamed

WHAT DOES IT MEAN FOR YOU?

Zet Universe will now remain responsive after importing large folders. Yes! This might sound strange, but by using really important and relevant date values, we've been able to significantly improve our product's responsiveness during heavy data processing (full-text indexing, keyphrase extraction, etc.).
"Created this item" will now show the time the file was created. This also means that if file was copied from another destination, Zet Universe will compare it's LastModified and Created dates, and pick the older one as "Created" date.
Picture
"Created this item" will now show the time the file was created.
Items will be now grouped by their real last modified values in the Navigation Pane. Consider this: an old project space that wasn't touched for a year. Once we've imported it into Zet Universe, all its files have been indexed, but it's clear that items haven't changed since that year ago mark (August 4, 2014). 
Picture
Real Created and Last Modified values are used to sort and group items in the Navigation Pane.

WHAT ARE THE OTHER BUG FIXES?

Well, we've made a lot of small bug fixes. Most of these fixes are related to better handling of situations where Exceptions were caught by the system. Here's a list of a few of such cases:
  • Zet Universe will now better index PDF files. We are using SyncFusion Essential Studio (for startups) for PDF thumbnail extraction (so that we could avoid using Adobe Reader APIs for that); however, there was a confirmed bug in the version of the dll we packed with Zet Universe. Starting today, we've moved to the latest version of SyncFusion Essential Studio which includes the bug fix for PDF thumbnail extraction.
  • Zet Universe no longer updates contacts in Dropbox each time it requests Dropbox web site for those who edited the given file. Prior to this build, Zet Universe issued checks for updated contact data after checking for each file, and in case you've imported a large chunk of files, Zet Universe would waste a lot of time (and place in the log) for checking contacts for updates.
  • Zet Universe no longer tries to write its log to the Windows Event Log. Initially, we expected you to install Zet Universe with admin rights. However, as we've moved to Squirrel for Windows, and it's amazing update experience, we've found out we have no good reason to require admin rights except for creating own Windows Event Log. Given that we write log to %appdata%\Zet\Logs\ anyways, it's not necessary to use Windows Event Log. Before this build, we had to make the check for the Windows Event Log, and handle an exception in case the Log is inaccessible.
  • Sticky Notes account is now accessible via Settings --> Accounts. Indeed, this is a really small thing, but we've managed to ship several builds of Zet Universe w/o any way to see Sticky Notes account. That's sad, there's nothing there to edit (for now), but for the sake of consistency, we've added it to the list. 
Picture
Sticky Notes Account available in Settings --> Accounts
  • System.UnauthorizedAccessException should no longer happen, and lead to a fatal crash of Zet Universe when trying to open Local Folders pane to start tracking a folder. This was a rather sad case, when Zet Universe tried to enumerate all folders, and then obtain their metadata, which for inaccessible folders was impossible to do, and led to a fatal crash. We've fixed that. 

WRAPPING UP

Ok, so, this time we've not only fixed at least one of the bugs we've introduced last time, but we've almost made a very important change to the entire user experience of Zet Universe.

We hope... No, we pray that you'll just love these changes! Because we're positively in love with this change!

And now, we are very thankful to you, our Insiders: without your feedback (hey, guys, you've given us almost 700 pieces of feedback, directly or indirectly; this is more than we've logged in our issue tracking before the Insider Preview Launch this July!), it would be impossible for us to make Zet Universe better for you.

Now, enough of this mutual admiration society. As we continue working on the SDK and new features, we encourage you to update to this build as soon as possible, as well as check out our updated product plan (check out the "What's Next?" section), and let us know what you want via the Feedback Forum, and use the Facebook group to connect with other Insiders for help and tips on problems you hit.

Thanks for your continued support!!!

Daniel and the team
]]>