iOS Syncthing Client (or Rust in the Wild)

It took a bit of time, but finally fsync is available in the AppStore.

From a user perspective it’s a pretty simple iOS client for syncthing. If you never heard about syncthing and you ever needed a self-hosted Dropbox you should give it a try. It’s very nice piece of software, it runs almost anywhere (Linux, *BSDs, Windows). It’s open source and written in Go so if you want to practice Go language – you’re welcome to contribute.

Right now fsync works in mostly “read-only” mode, i.e. you can view files, you can delete them, but there is no way to edit. This is gonna change soon.

From a technical perspective it’s probably one of the earliest iOS applications, which are partially written in Rust.

There is a lot of things to talk about and I hope to provide more technical details in the following posts, while this one covers using Rust as a language for mobile development from my experience.

How it started

Being active syncthing user I was missing opportunity to access my files from mobile and decided to write an iOS application.

Initially I’ve started writing core code in C, got a first working prototype and understood that I’m spending too much time dealing with managing memory, ownership and so on. I’ve checked available options:

  • C++
  • Rust
  • Swift
  • Objective-C

C++ was dropped immediately – I haven’t written in it since university, it changed a lot… and I wouldn’t like to invest my time into it, even if gives more advantages in short-term.

While technically it was possible to use Objective-C, it’s not good enough for low level stuff and there is a lot of low-level stuff in syncthing client.

Swift looked very promising, but it was so immature that even Apple “fanboys” were against choosing it for any serious code.

The problem I saw was that although there is an official Android client, it’s basically re-packaged syncthing. Which means it wasn’t written with mobile in mind, where being memory/traffic/battery friendly is crucial.

You get were I was heading – I’d be happy to have a cross-platform code written for mobile. And the only option in this case was Rust.

Maturity

There were moments when I hated Rust – it was a hell of a ride to the API stability. Initially it didn’t feel like a big deal (“ohh, that’s just a couple of changes here and there”). But… Little strokes fell great oaks and a couple of times I was on the edge, considering to rewrite it again in C (or C++ – yep, it’s a huge monster, but it’s also a well known + stable monster).

One of the hidden problems caused by instability was a desire to keep breaking surface as small as possible. Practically it meant ignoring existing libraries and re-inventing half-baked versions just to have a control on when and how to update it.

It was a dark time and I’m glad it’s finally over.

While language and std lib are stable now, there is still a huge necessity in a better documentation. A lot of changes have happened with no announcements and no discussions so if you were always on GitHub, monitoring issues everyday – you’re probably fine. But if you weren’t because you had to write your own code – I bet a lot of that code is already “rusty” and cries for modernization.

What was bad

At some point I was very disappointed, but I have to admit it was caused mostly by wrong expectations and naive estimations.

  1. I didn’t expect it’ll take so long to stabilize.
  2. Rust can give you exceptionally granular control of memory allocations, but it’s a manual control. You’ll pay with a longer development cycle, you’ll pay wit much more complex code and less flexibility.
  3. Rust is slow to compile and that discourages a lot, especially if you’ve got a minute to hack on an old laptop and… Boom… Half of the time is wasted in wait for compiler. More than a minute for building ~7K LOC? Yeah, I know it’s an old laptop, but that’s hardly an excuse, considering Clang can handle much bigger codebase faster.
  4. Code-management tooling (navigation/refactoring) is immature. This should improve over time with language stability, but I can’t imagine how long it will take to get something like AppCode/CLion.

As you can see only last 2 reasons are related to the language itself and I hope that at least compilation time will be addressed in upcoming 1.1 or 1.2.

What was good

Before Rust I’ve been developing in higher level languages for quite a long time. As a result, there were a lot of things I didn’t even consider – they were hidden, the choice was already made for me. The first impression after using Rust was “ohh, there are so many things I have to decide on my own now!”.

For me personally it was a refreshing experience and also a reminder that there is no way to get performance if you don’t keep it as a primary goal all the time.

Probably more than the language itself I liked the direction it takes in encouraging good engineering practices – you get built-in testing/benchmarking support and a pretty decent documentation generator. Want to go further? Mix them – and you’ll get a bullet-proof documentation, which will fail your tests if you’ve forgot to update it after an API change.

And yet there is also Cargo for getting repeatable builds out of various dependencies. Std lib was significantly reduced on the way to 1.0 due to cargo. What was not necessary or was controversial was extracted out of tree to a separate crate1.

That gut feeling of rock-solid foundation becomes even stronger after familiarizing with Rust’s development process. In short – there is a bot, which runs tests on approved pull requests and merges into master only on success. Sometimes it’s annoying, sometimes it’s slow, but you’re sure that at any point of time nothing is broken2.

iOS development

One of the most interesting questions is if there is a place for Rust in iOS development.

In most cases if you’re developing for iOS/OS X – you’d better use Objective-C and/or Swift. They provide much better integration with native frameworks and higher level to work on.

Still, Rust can shine when:

  • you need a cross-platform code
  • you target to have a very low footprint
  • you can either isolate code enough or provide automatic bindings to/from “native” code

I’d say that the last one is the real bottleneck.

A lot of iOS/OS X frameworks have a pure C interface3, 4 and therefore can be used from Rust directly5. In general it’s easy enough to access Objective C objects either6, but it’s hard to exploit that without loosing portability. So… you can’t move things down to Rust7 – you have to provide smoother interaction between “core” and “UI” by other means.

When Dropbox started cross-platform C++ development, they had the same problem and they solved it8 in Djinni – an automatic binding/proxy generator which bridges native code with cross-platform code. If someone writes a generator for Rust – it can make a huge difference9.

Another way to workaround this problem is to have a pretty isolated Rust core, which acts as a server and can be accessed through a well-known protocol. It means there will be additional costs, definitely higher than automatic data marshaling performed by Djinni. Whether it is acceptable or not depends a lot of data flow in the application.

As a conclusion – unfortunately for mobile cross-platform development Rust is pretty limited nowadays and works good only for specific use cases. A bright side though is that it’s a limitation which can be addressed by tools.


  1. crate is a synonym for a package

  2. unless you’re out of luck and provide by yourself support for a platform which hasn’t official buildbot.

  3. CFFoundation for example provides access to all native data structures.

  4. Steven Sheldon got into GCD and you can even provide Rust blocks to Objective-C code.

  5. unsafe wrappers can be generated by bindgen, it takes a bit more time to get safe wrappers though.

  6. see examples in rust-objc, rust-objc-foundation from Steven Sheldon and rust-cocoa from Servo.

  7. with the only exception for platform-accelerated APIs.

  8. slides from the talk with more details about goals and motivations.

  9. there is a chance that Dropbox writes generator for Rust as they use it internally, although I doubt it.

Write Only: Setup

In earlier post I’ve written about switching to write-only mode inspired by Joe Nelson. This time I’ll provide a bit more details on why and how.

Considerations

As I’ve already mentioned, for me “write-only” mode is not about avoiding ephemeral things (is there anything “real”?) or being disconnected – it’s all about being proactive instead of reactive, this is the only way to concentrate on producing things.

It definitely requires a special environment and I’d say that the most important things are organizing space and time.

By time I literally mean time – it’s impossible to create something you’re proud of without iterating, prototyping, throwing bad attempts, taking time to recover and continue looping.

By organizing space I mean a bit more complex concept. The act of creation is always a path from entropy to a structure, from chaotic bits to organized bits. As your knowledge is what allows you to create, it’s kind of obvious that you should concentrate on converting information to knowledge.

What’s the difference between the two? Connections! It’s all about connections – they form the structure, they form the way you think, what you can think and what you can’t think of.

The thing with connections is that you can see them much easier in “big forms” – no matter what we’re talking about – be it music, be it an idea, be it a life story or anything else.

Tell me how you spent yesterday – I’ll know a couple of facts about you. Tell me about your whole life – I’ll find patterns, I’ll make my conclusions about people in general and your character in particular.

Imho, the most wrong thing about social media and news is that being “small forms”, they just overload you with disconnected bits of information. You can’t use it, but it still occupies (or rather pollutes) your mental space. The space you could use for creating.

That brings an interesting hypothesis: the form of information you consume is much more important than how old and “time-proven” it is (I feel it’s “weird” English, let me know if you have an idea how to reformulate it). You know, it’s always a big surprise what exactly is the last drop for an “Eureka!” moment, but it always is about filling a “last connection” in already highly organized “model”.

One of the reasons why “big forms” are usually much useful is that they require a stronger structure. It means author has to work much harder in both discovering connections and in explaining them to the “reader”.

The simplest lesson I can take from this observation is that “more you write – better you understand”.

This works for other communication channels too – I bet every developer knows how effective rubber duck debugging is. What makes writing so special is that there are no time limits and that helps a lot.

Practical setup

Considering all above, I’ve formulated the following thesises:

  • big forms should have preference over small forms (both for consumption and for production)
  • big forms require huge time chunks
  • communication shouldn’t be blindly cut – it’s all about sharing experience, it should be reshaped if required
  • time spent writing down thoughts should be increased
  • thoughts should be shared publicly and being non-native English speaker is not an excuse
  • there should be no isolation from new information
  • new information should be dosed more than “time-proven”
  • it’s preferable to have new pieces close to the current expertise – it’s much easier to evaluate them and turn into something useful today

After a bit of tinkering, here is a part of my current setup:

  • turned off notification sounds/vibration on the phone, they split time into smaller chunks and destroy concentration on big forms
  • never take phone to the bed – smartphone creates a temptation to start another round of Hacker News/Reddit/Twitter. It allows to sleep earlier and wake up disconnected
  • no news until midday
  • news and browsing are actually about quick skimming to understand if it deserves to be marked
  • marked pieces are queued (in my case to Instapaper)
  • I have a time limit to spend on queue => it’s another filter to pick important things, letting noise to be drown
  • no automatic mail retrieval – email is handled in batches every couple of hours
  • earlier I gathered email from different accounts into one. I’m reverting it back to have a separate accounts for different tasks. This + manual mail retrieval allows to avoid temptation on answering personal email during work time and vice versa – I literally can see only emails of “current working context”
  • I’m also experimenting with GTD-approach to email like in Google Inbox and similar products
  • no more short inline comments
  • if I want to suggest something – I have to write a spec, instead of throwing out raw idea
  • I have to ressurect my blog, it was actually never active but only because I’ve always thought there was nothing I could share.

The only missing solution so far is IM, to be precise Slack-based work communication. In most cases my response time raised significantly (up to 25 minutes for a reply), but so far that works fine.

Preliminary results

I’m practicing this setup a bit less than 2 weeks and so far I have the following observations:

  • it’s really hard to get off the news hook – sometimes it feels like I’m blinded
  • on the bright side I’m much more connected to everything I work on
  • I’m more interested in long-term results
  • there are no temptations in trying one more new shiny thing – I don’t know about them :–)

Write Only Mode

By accident I’ve read today about Joe Nelson’s life experiment.

It’s a really inspirational post for me as it resonates a lot with what I’ve though about recently. I can argue a lot of minor details (because “but the great man is he who in the midst of the crowd keeps with perfect sweetness the independence of solitude”), but I can’t agree more with general direction the author takes.

I believe that one of the most limiting factors for current civilization is a culture of consumption. We have too much to consume and our uncontrolled consumption as a result “consumes” our time and available resources, leaving no opportunity to produce and create something.

It’s especially true for developers as we’re overhelmed by available technologies/frameworks/libraries/languages and are interested in trying “one more” shiny, just released thing, which is awesome and just got to HN. I’ve personally done that mistake dosens of times and I’m gonna tell about the recent one in the following posts.

To clarify: it’s not about sticking forever with old tools. It’s about clearing as much space/time as possible for generating new things/thoughts/ideas/products.

I’m not sure I’m ready to formulate it more clearly now, so I’ll start a REPL with my own experiment – instead of limiting myself, I’m going to make sure that I’m in “write only mode”.

What’s the write only mode? It’s the mode where I produce at least as much accessible output as I consume in my expertise areas. Preferably it should be a publicly available output.

The idea here is that it’s actuallly impossible to stop consumption by itself – it’s a dead end, especially in non-expertise areas. But what really makes a difference is what comes out of consumption – my only joy or something others can also use.

Continuation: considerations and setup

Rust for iOS: Status Update

It’s been a while since last status update, but I hope it was worth waiting. Rust is now completely ported to iOS: all architectures for device/simulator are supported. Since green threads were removed, adding support of arm64 was actually almost straightforward. Special thanks this time goes to Will Glozer who was debugging remotely. Fortunately for our sanity, there were just a couple of minor issues.

Although all changes have been landed to master, cross-compiled libraries aren’t distributed with nightly, so in case you’re interested in trying it out, you can either build it from source (instructions) or use a pre-built dist.

As earlier, the main usage sample is located here. It still uses makefile based approach and I hope to update for cargo a little bit later.

Rust Docsets for Dash

Since I’m tracking Rust nightly and documentation from the last release was quite outdated, I’ve wrote a quite simple (at least I think so) docset generator for Dash (wonderful piece of software).

Here is the link to feed which is updated once new nightly released.

Once it was complete, I’ve understood that I’d like to see docs for my libraries in Dash as well and here is what I’ve got: rust-docset – it is a Python script which allows to generate Dash docsets and feed from any Rust docs.

It requires a pretty simple TOML manifest to be available, like this one:

 1 [docset]
 2 name = "Piston"
 3 bundle_id = "piston"
 4 version = "0.0.1"
 5 type = "docset.rust"
 6 doc_dir = "../doc"
 7 out_dir = "doc_out"
 8 
 9 [feed]
10 base_url = "http://s3-us-west-2.amazonaws.com/net.vhbit.rust-doc"
11 upload_cmd = "./upload.sh"

and upload.sh might look like this:

#!/bin/sh

s3cmd put -P $1 s3://net.vhbit.rust-doc
s3cmd put -P $2 s3://net.vhbit.rust-doc

While it started as Rust only project, I believe it could be relatively easily programmed to work with documentation generated by other tools.

Rust for iOS: Merged Into Master

Current state

About a week ago it was finally merged into Rust master branch which is really great as it is much easier to work on smaller changes now.

Here are instructions on how to build it. So far there is no official iOS buildbot and therefore it might broke at any moment, in this case I recommend to use a original fork which could be considered a “stable” branch, tested to compile and work both on device and simulator. It might lag a bit from the master but not that much.

There is still one issue in LLVM which sometimes crashes compilation process. Patch was submitted but considering previous experience I wouldn’t expect it to be landed in less than a month.

I’ve started work on Arm64 port but it definitely will take much more time considering I have no test device.

Rust for iOS: Progress Report

Although there is no public indication, work on porting Rust to iOS continues, moreover it is going to the final stage. So far I’ve got all required pieces working:

  • fixed issue with mutexes crashing libgreen/rustuv runtime
  • got a working patch for LLVM to support segmented stacks
  • got working unwinding (another patch to LLVM compiler-rt)

In general, all major features are working now, providing first-class support of all Rust safety. There definitely might be small hiccups here and there but I believe they could be solved step-by-step, once the major part will be upstreamed to Rust master branch.

I need some time to clean up the code (as it has a lot of experimenting branches) and once patches to LLVM and compiler-rt landed – the last step will be rebase on current master and issue an PR.

I hope it will be finished this week.

Rust on iOS

Porting process

Finally got working Rust cross-compiler for iOS so now it is possible to create static libraries which could be used from Objective C. For now only armv7 architecture is supported (although there are upcoming commits in “try” branch which will update LLVM version with ARM64 support right from Apple).

These were main pain points:

  • Slow full rebuild… Yep, I know, I know, 3 stage bootstrapping to be extremely optimized but still… it was especially painful on initial stage while experimenting with built options and LLVM patching.

  • TLS isn’t supported on armv7 so it should be emulated. Initially I’ve started adding it to LLVM and with help from Tim Northover and Dan Olson even got it working. (BTW, Dan knows a lot about this topic as he got a working D compiler for iOS, so if you’re interested in modern compiled language for iOS – you can try it too). Just to discover that actually Rust has built-in TLS “emulation” support (actually using the same technique as in LLVM patches)… Grr, hate myself for not checking this before messing with LLVM.

  • There were absolutely crazy crashes hard to debug because some kind linker errors/misconfiguration when functions were resolved correctly but all the data was relocated incorrectly and contained garbage.

  • There were mystical crashes in hash function which automagically disappeared once I’ve increased optimization level.

  • The final show-stopper was happening in runtime finalization routine – it always crashed on misaligned data during mutex locking. The reason was that mutex opaque data was copied as a plain data. It wasn’t the case on desktop OSes but seemed to be a big difference on iOS. There is a simple fix for native runtime and should be similar fix for green/rustuv.

What works

  • armv7 only (ARM64 port should be quite easy to introduce)

  • it is possible to compile standard libraries and use them

  • “native”-based runtime

  • creating static libraries which could be used from iOS applications

Known issues

  • green/rustuv runtime crashes because of incorrect mutex implementation

  • segmented stacks aren’t working

  • it might require additional manual linking

  • no support for simulator yet

  • it only works on common compiler flags. It is critical to have exactly —opt-level 2 (or -O) as both decreasing and increasing opt level may cause a lot of unexplainable issues

Shut up and take my money Build instructions

clone and create build dir

git clone git@github.com/vhbit/rust.git
cd rust
git checkout ios
mkdir build_ios
cd build_ios

configure

../configure --target=arm-apple-ios

build

make

or (for 4 core CPU)

make -j4

There is a sample project with instructions on how to integrate.

Thanks

Special thanks to Tim Northover (ARM TLS), Dan Olson (ARM TLS), Alex Crichton (Rust internals) and my wife for being patient.

Joe Armstrong об ООП, Concurrency

Две части замечательной беседы с Джо Армстронгом (создателем erlang'а) об объектно-ориентированном программировании и почему оно скоро упрется и вынуждено будет смещаться в сторону функционального.

Один из интересных тезисов – это то, что сейчас нас начинает поджимать железо, если раньше мощности росли в количестве гигагерц, то сейчас все идет за счет увеличения числа ядер. А это провоцирует на переход к concurrent-based программированию (есть, кстати, нормальный русский термин для этого?), где императивщина в чистом виде начинает загибаться из-за возрастающей сложности разнообразных локов и защит.

Материал 2007 года и этим вдвойне интересней, т.к. тренды, о которых говорит Джо, из гипотетических становятся все более и более ощутимыми.

Переопределяем метод класса в рантайме

В кои-то веки потребовалось в рантайме переопределять методы класса. В документации сразу нашлась подходящая функция:

BOOL class_addMethod(Class cls, SEL name, IMP imp, const char *types)

единственный аргумент, который может смутить – это types, который задает типы аргументов и возвращаемого значения в закодированном текстовом виде. Нужные значения можно посмотреть либо в документации, либо исппользовать @encode.

Вторая тонкость оказалась в том, что эта функция добавляет (или переопределяет) метод экземпляра, а что же делать если нужно заменить метод самого класса?

Правильно, получить метакласс с помощью object_getClass(ourClass) и заменить его метод как обычно :)