Signals Demo

Your first name is: {{ firstName$ | async }}

) and knowing what the component is (

). The author writes: "Yet, the bigger and the more complex a component gets, the less obvious it is to know what class name maps to what element on the screen, or what it looks like." But then they include an example where it's absolutely clear which components do what:

The Shining

His breath stopped in a gasp...

Stephen King

Stephen Edwin King ...

Website Twitter

I think this again shows the difference between programmers and designers: the code above is crystal clear to a programmer, so if a programmer is writing the CSS, then there's no need to change anything. The author seems to be a designer hell-bent on knowing exactly what the page will look like without actually showing it in a browser. I wish they'd included the version with utility CSS ... it would have been a giant block of unreadable code, doubled in size with class names. [Don't Change Anything] The author makes a good case for theming using CSS variables, which can be applied "at runtime" in the browser. The solution to theming with utility CSS turns out to be ... making semantic styles instead of precisely named styles. So...not utility CSS. The author references a few other articles, one of which is "Kiss My Classname" by Jeffrey Zeldman , which eloquently argues that there is nothing to change. He instead argues that developers and designers should use a visual style guide. "I don’t believe the problem is the principle of semantic markup or the cascade in CSS. I believe the problem is a dozen people working on something without talking to each other. "Slapping a visually named class on every item in your markup may indeed make your HTML easier to understand for a future developer who takes over without talking to you, especially if you don’t document your work and create a style guide. But making things easier for yourself and other developers is not your job. And if you want to make things easier for yourself and other developers, talk to them, and create a style guide or pattern library." "The present is always compromised, always rushed. We muddle through with half the information we need, praised for our speed and faulted when we stop to contemplate or even breathe. (Emphasis added.)" -------------------------------------------------------------------------------- [1] It's also almost two years old, but still seems to describe the state-of-the-nation in utility/atomic CSS. [1] Another article they referenced was "CSS Utility Classes and "Separation of Concerns"" by Adam Wathan and it's even longer. It's almost a jeremiad with the seeming intent of breaking the reader down with a flood of words. I could only skim it, but it seems like these people are styling without programming: that is, some of the utility classes and even the slightly semantic ones they use could very easily be written more cleanly if they just used component-local styles. For example, this is completely unnecessary with local styles, because you don't have to worry about specificity biting you in the ass:

Stubbing ...

In this quick blog post and screencast, ...

In another article "On the Growing Popularity of Atomic CSS" by Ollie Williams , the author mentions that they're addressing "n a mixed-ability team, perhaps involving backend developers with limited interest and knowledge of CSS". I didn't have the energy to finish that one either, because a skim indicated that it repeated a lot of what was in the article I did read. ]]> https://www.earthli.com/news/view_article.php?id=3825 2019-10-17T14:42:13+02:00 https://earthli.com/users/marco

https://www.earthli.com/news/view_article.php?id=3823 2019-10-17T13:27:26+02:00 https://earthli.com/users/marco

F7. This is helpful if you moved code and want to compare two chunks that the standard diff no longer sees as being comparable [Merging] I could live without the Beyond Compare differ, but not without the merger. [image] * The 4-pane view shows left, base and right above as well as the target below, with the target window being editable. Each change has its own color, so you can see afterwards whether you took left, right or made manual changes. * The merge view includes a line-by-line differ that shows left, base, right and target lines directly above one another, with a scrollbar for longer lines. * The target view is color-coded to show the origin of each line of text: right, left, base or custom edited. * BeyondCompare makes a smart recommendation for how to merge a given conflict that is very often exactly what you want, which means that for many conflicts, you can just confirm the recommendation. * SmartGit has two separate windows for base vs. left/right and right/left vs. target. Long lines are really hard to decipher/merge in SmartGit [Integrate Beyond Compare into SmartGit] To set up SmartGit to use Beyond Compare 1. Select Tools > Diff Tools 1. Click the "Add..." button 2. Set File Pattern to * 3. Select "External diff tool" 4. Set the command to C:\Program Files (x86)\Beyond Compare 4\BCompare.exe 5. Set the arguments to "${leftFile}" "${rightFile}" 2. Select Tools > Conflict Solvers 1. Select "External Conflict Solver" 2. Set File Pattern to * 3. Set the command to C:\Program Files (x86)\Beyond Compare 4\BCompare.exe 4. Set the arguments to "${leftFile}" "${rightFile}" "${baseFile}" "${mergedFile}" [Update March 11th, 2021] I was testing the Git support in Visual Studio Code and ran into a somewhat surprising limitation. For those that use IDE Git integration without an external tool, this would be a pretty disappointing message. What do you do then? [image] -------------------------------------------------------------------------------- [1] In "Git: Managing local commits and branches" and "Programming in the moderncurrent age" [1] I am in no way affiliated with SmartGit. [1] I am in no way affiliated with BeyondCompare. ]]> https://www.earthli.com/news/view_article.php?id=3822 2019-10-17T07:38:00+02:00 https://earthli.com/users/marco

https://www.earthli.com/news/view_article.php?id=3743 2019-05-19T17:15:28+02:00 https://earthli.com/users/marco

. I think the product itself sounds interesting, but that the author uses unnecessarily misleading arguments. From what I gather, Svelte is a compile-time reconciliation generator for JSX/TSX components.... ]]>

. I think the product itself sounds interesting, but that the author uses unnecessarily misleading arguments. From what I gather, Svelte is a compile-time reconciliation generator for JSX/TSX components. This pre-calculated generator applies changes to the DOM without needing a virtual DOM and without real-time diffing or reconciliation. That is, instead of having real-time calculation, with possible performance hits [1], the app benefits from having all possible state changes pre-calculated and ready to apply immediately and quickly. This all sounds pretty good, I think. I'm definitely going to take a look at the more-advanced tutorials. [2] However, the author wasn't happy with just presenting his product, but seems to need to mischaracterize why products like React abstracted away from the DOM in the first place. He tells us that the virtual DOM was always slower than manipulating the DOM. But that isn't the claim React makes. React helps users avoid common performance pitfalls in the model of programming that it replaced -- it never claimed to be the final word in performance optimization. It's clear that something like Svelte -- if it can cover all the needs of an app -- is faster than maintaining a virtual DOM. But that product isn't what React replaced. React replaced products written in jQuery. React brought an asynchronous frame-based renderer to the web (something that products like WPF have had for decades). It brought us type-safe views (when used with TypeScript) and taught us about the advantages of immutable data structures. "He stands on their shoul-ders, then implies that they were idiots for not having been taller." The author characterizes the notion that a virtual DOM is faster as a "meme". This is silly and imprecise. It is true that React will be more efficient than most hand-coded web sites of a typical level of complexity. jQuery sites tended to teeter and collapse under their own weight. They were unmaintainable and very difficult to optimize without nearly rewriting them. React sites, on the other hand, are modular in nature and the library includes several standard patterns to apply and measures to take to optimize these components. It's not always easy, but it's better than it was in the old days. And there are solutions in React to performance issues. The users must follow patterns and use the APIs correctly. That's the way it is in every framework or library. Some libraries offer less leeway for users to screw up performance in the way that they shape their APIs. Sometimes the API surface goes too much in that direction and ends up handcuffing users. That is, users can't write what they want to write in a way that feels natural because the pattern they prefer wouldn't perform well under their framework. Instead, the user must change how they think about writing apps just to use the framework. This isn't necessarily a bad thing, but is definitely something to consider. It's possible that Svelte offers all of the advantages of React with even more flexibility and less opinion. React -- and its companion Redux -- was always about being very declarative about state and changes. There is no magic, even the reconciliation algorithm is very predictable. There are other approaches, like MobX, which users claim "does the right thing" with state changes, even if the user fails to declare dependencies as clearly as React would have required. [3] I imagine that Svelte is going in this direction as well. The claim I think that Svelte is making is that users can write code that feels more natural without changing their paradigm to match the framework. That is, Svelte must have some rules for which state the compiler observes and pre-compiles, but the claim is that it's much more flexible and forgiving than React's "straitjacket" (my word). [4] He goes on to say that React acknowledges its own slowness by giving the user control over shouldComponentUpdate. This is a silly argument again. It's arguing that React bamboozled people in 2013 by convincing them to use their framework instead of a library that the author purports is faster but that he only started in 2017. There is honestly no need for this kind of bullshit. If your library offers advantages over React, describe them and let them speak for themselves. There is no need to rewrite the whole history of a product that quite clearly inspired your own, pretending that the authors of your own framework's inspiration are your inferiors because they failed to leap directly to the concepts outlined in your library. He stands on their shoulders, then implies that they were idiots for not having been taller. Through all of this fluff, it took to about 3/4 of the way through the article to find out that Svelte generates update code at build time. I would have been much more intrigued had the author led with that. Now, I'm going to be suspicious of everything about this framework because the author went to such lengths to bamboozle and oversell me. He seems to want me to think I've been a fool for having used React in the first place, when his framework has been waiting for me all along, since all the way back to sometime in 2018. [5] But he waits until the very last paragraph to explain what Svelte actually is -- even though he's been comparing it to React the entire time. It's a good description: "It's important to understand that virtual DOM isn't a feature. It's a means to an end, the end being declarative, state-driven UI development. Virtual DOM is valuable because it allows you to build apps without thinking about state transitions, with performance that is generally good enough. That means less buggy code, and more time spent on creative tasks instead of tedious ones. "But it turns out that we can achieve a similar programming model without using virtual DOM — and that's where Svelte comes in." This is a much fairer characterization of the two libraries: they both base on a very similar model -- one that React did a tremendous amount of legwork in establishing as an attractive approach in people's minds -- but that Svelte goes a step further to improve the reconciliation mechanism, moving it from runtime to compile-time. Svelte's improvement could be a highly welcome one, but it's incremental, not revolutionary. That's wonderful! But it's actually even more wonderful than his article indicated, because I actually don't have to learn anything to work with Svelte instead of React. I can work pretty much the same (Svelte doesn't have hooks [6] because it seems it doesn't need them) and just kind of "drop in" Svelte instead of React and have better performance, even in places where I'd never noticed I might have had problems. That is, with Svelte instead of React, my app will be overall faster because performance no longer suffers from "death by a thousand cuts", as the author puts it. Despite the author's overzealous mischaracterizations and attempts at hot-take marketing, I'm still going to check out Svelte. -------------------------------------------------------------------------------- [1] Such performance hits are often irrelevant, as even the author admits. However, if you can choose between two frameworks, one that punishes you with bad performance and another that optimizes instead, you would choose the more forgiving one, if there are no other downsides. [1] Here I'm going to lead with my conclusion so as not to follow in the author's footsteps, using a style that I spend the rest of this article upbraiding. [1] I'm not sure what MobX 5 is up to or what introspection it offers into the web of observables and dependencies in a more-complex application, but older versions of the library were not easy to debug when performance problems arose. From what I've read from users, things have gotten much better, but I'm still inclined to think that React's declarative approach suits me better -- it's easier for me to apply well-established patterns in my own code rather than trying to figure out how to appease the MobX black box. Again, things may be different now than in earlier versions. I'm open to taking another look at MobX. I'm also not sure how Svelte and MobX compare: MobX requires users to indicate that state is "observable" before it manages it, whereas I assume Svelte determines for itself which state-transitions it should track. [1] Update January 2022: In going through the tutorial available today, you're very quickly introduced to reactive declarations to help Svelte determine which compound expressions should be "watched" for changes to sub-elements. That is, if you declare simple variable, any references to it in view code will be automatically updated, but if you derive another simple value from it and observe that value, it only updates when the derived value is updated directly. This is unlikely, as the derives value presumably implements an algorithm of some sort and should never be directly changed (i.e. it's a calculated property in the parlance of other frameworks. For example, given the following code, let count = 0; doubled = count * 2; Any observers (i.e. embeddings in a view) of the value of doubled will not be updated when value changes, even though the naive interpretation of a JavaScript developer would be that of course it changes. In order to get the desired effect, you must make it a reactive declaration with $:. For example, let count = 0; doubled = count * 2; This is perfectly fine, but is an example of how the "you don't have to do anything to make your JavaScript work naturally, unlike smelly React", is overselling the advantage. Missing reactive declarations will cause an app to not work as expected just as must as a missing useState() does in React. [1] It's hard to tell when it was released because there are 254 releases, one nearly every day and I didn't feel like hitting "older" in GitHub for 30 pages or cloning the repository just to get a better look at the log. [1] The author disparages hooks, saying that they are even worse for performance and linking to a tweet with the words "with predictable results". The tweet complains about atrocious performance because of constant reconciliation and rendering -- but a dozen answers down is the answer: the original poster failed to tell the useEffect() hook on which state it relied. That's kind of a rookie mistake -- in that framework. [7] I understand that Svelte claims that it doesn't need these hints in order to be able to determine at compile-time when a piece of code needs to be executed because the state on which it relies has changed. React is declarative and requires help from the user whereas presumably the selling point of Svelte is that this user would have wasted less time improving performance and more time focused on application logic because Svelte is smart enough to do all this for you. I personally think that this sounds awesome and that it is an admirable goal, but have my doubts that Svelte doesn't also impose its own set of limitations on what kind of state transformations you can do that the compiler can actually detect. That is, React provides an API with which callers can "help" the reconciliation algorithm avoid work. Svelte claims that this isn't necessary, but I'm going to guess that there are rules for how sophisticated state changes can be before the Svelte compiler no longer detects them. In that case, what does Svelte do? Fall back to using a React-like virtual DOM to reconcile changes? Or just not update when the user expects? Or just fail to compile, spitting out an error indicating what the user should do to fix them issue (my personal favorite)? [1] Update January 2022: It's also a mistake that IDEs will help you avoid. WebStorm issues a warning if you rely on state in an effect that you have not listed in the dependencies. ]]> https://www.earthli.com/news/view_article.php?id=3487 2019-04-08T09:38:17+02:00 https://earthli.com/users/marco

* Don't forbid mocking in integration tests and don't force mocking in unit tests. * In fact, stop worrying about whether it's a unit or an integration and just write useful tests that prove useful things about your code. * * Don't get obsessed with automating everything. * Get the low-hanging fruit first, and leave the rest to manual testing. * See where you stand. * If you haven't automated enough, iterate until done. 🔄 [Tests should be useful] We never want anyone in a team to get the impression that we're writing tests just to write tests. We write tests because they help us write better code and because it feels good to be able to prove that something that was working continues to work. You should feel more efficient and productive and feel like you're producing higher-quality code. * Tests should confirm use cases * Tests should prove something about your code that you think is worth proving. * Tests should confirm behavior that either is how the code currently works or how it should work. * Tests should help you write better code from the get-go. * Every bug that you need to fix is de-facto a use case that needs a test. [Code Coverage & Reviews] How do you know when there are "enough" automated tests? Don't get distracted by trying to achieve a specific coverage percentage. The most important thing is that the major use cases are covered. If software is stable and there is "only" 40% test-coverage, then maybe there is a lot of code that rarely or never gets used? In that case, you might want to think about removing code that you don't need rather than to waste time writing tests for code that never runs. New code, though, should always have automated tests. A code reviewer should verify that new functionality is being tested. [Types of tests] Unit Cover a single unit, mocking away other dependencies where needed. Useful for verifying simple logic like calculated properties or verifying the results of service methods with given inputs Integration Cover multiple units, possibly mocking unwanted dependencies. Useful for verifying behavior of units in composition, as they will be used in the end product. The goal is to cover as much as possible without resorting to more costly end-to-end tests End-to-End Also called UI Tests, these tests verify the entire stack for actual customer use cases. Very useful, but generally require more maintenance as they tend to be more fragile. Essential for verifying UI behavior not reflected in a programmatic model. Can work with snapshots (e.g. error label is in red) [Approach] The article "Write tests. Not too many. Mostly integration." describes a pragmatic approach quite well. Instead of the classic "testing pyramid", it suggests a "testing trophy". !"image.png" This style of development has the following aims: 1. Verify as much as possible statically, with linting and analyzers 2. Make integration tests cheaper because they prove more about your system than unit tests 3. Prove as much as possible outside of end-to-end tests because they're expensive and brittle [Analysis] "Remember that everything you use has to work both locally and in CI." [Static-checking] A project should include analyzers and techniques so that the compiler helps make many tests unnecessary. For example, if you know that a parameter or result can never be null, then you can avoid a whole slew of tests. Developers should only spend time writing tests that verify semantic aspects that can't be proven by the compiler. [Null-reference analysis in .NET] The .NET world provides many, many analyzers and tools to verify code quality. One of the most important things a project can do is to improve null-checking. The best way to do this is to upgrade to C# 8 or higher and enable "null-reference analysis" . The "default language for .NET Framework is going to stay C# 7.3" , but you can "enable null-reference analysis for .NET Framework" quite easily. Another option is to use the "JetBrains Annotations NuGet package" , which provides attributes to indicate whether parameters or results are nullable. The preferred way, though, is to use the by-now standard nullability-checking available in .NET. Doing neither is not a good option, as it will be very difficult to avoid null-reference exceptions. [Unit-testing] Unit tests are very useful for validating requirements and invariants about your code. These are the easiest tests to write and will generally be the first ones that you will write. A requirement or an invariant may be specified in the story itself, but it can be anything that you know about the code that's important. It's up to the developer and the reviewer(s) to determine which tests are necessary. It gets easier with experience—and it doesn't take long to get enough experience so that it's no longer so intimidating. [Unit-testing example] Just as a quick example in .NET, consider the following code, public bool IsDiagnosticModeRunning { get => _isDiagnosticModeRunning; set { _isDiagnosticModeRunning = value; _statusManager.InstrumentState = value ? InstrumentState.DiagnosticMode : InstrumentState.Ready; } } Here we see a relatively simple property with a getter and a setter. However, we also see that there is an invariant in the implementation: that the _statusManager.InstrumentState is synced with it. Using many of the "techniques described below" <#tools-and-techniques>, we could write the following test: [DataRow(true, InstrumentState.DiagnosticMode)] [DataRow(false, InstrumentState.Ready)] [TestMethod] public void TestIsDiagnosticModeRunning(bool running, InstrumentState expectedInstrumentState) { var locator = CreateLocator(); var instrumentControlService = locator.GetInstance(); var statusManager = locator.GetInstance(); Assert.AreNotEqual(expectedInstrumentState, statusManager.InstrumentState); instrumentControlService.IsDiagnosticModeRunning = running; Assert.AreEqual(expectedInstrumentState, statusManager.InstrumentState); } Here, we're using MSTest to create a parameterized test that, * creates the IOC * gets the two relevant services from it * Verifies that the state is not already set to the expected state (in which case the test would succeed even if the tested code doesn't do anything) * Sets the property to a given value * Verifies that the state is correct for that value We now have code that validates two facts about the system. Should something change where these facts are no longer true, the tests will fail, giving the developer a chance to analyze the situation. * Was the change inadvertent or deliberate? * Are the facts still correct? Does the test need to be updated? If you're addressing a bug-fix, though, you might be able to prove that you've fixed the bug with a unit test, but it's also likely that you'll have to write an integration test instead. [Integration-testing] Unit tests have their place, but they are far too emphasized in the testing pyramid. The testing pyramid comes from a time when writing integration tests was much more difficult than it (theoretically) is today. The "theoretically" above means that the ability to write integration tests as efficiently as unit tests is contingent on a project offering proper tools and support. One common complaint about integration tests vis à vis unit tests is that they run more slowly. Another is that they take longer to develop. Ideally, a project provides support to counteract both of these tendencies. To this end, then, a project should offer base and support classes that make common integration tests easy to set up and quick to execute: * Interacting with a database * Setting up a known database schema * Getting to a clean dataset * [Mocking]() the database * Mocking other external dependencies in a project (e.g. loading configuration from an endpoint, sending emails, sending modifications to endpoints) There are many different ways to solve this problem, each with tradeoffs. For example, a project can load dependencies in Docker containers, either created and started manually (see "Testing your ASP.NET Core application - using a real database" ) or even dynamically with a tool like the "Testcontainers NuGet package" . [Comparing Unit and Integration tests] A drawback to unit tests is that, while they can test an individual component well, it's really the big picture that we want to test. We want to test scenarios that correspond to actual use cases rather than covering theoretical call stacks. It's not that the second part isn't important, but that it's not as important. Given limited time and resources, we would prefer to have integration tests that also cover a lot of the same code paths that we would have covered with unit tests, rather than to have unit tests, but few to no integration tests. This, however, leads directly to... The advantage of a unit test over an integration test is that when it fails, it's obvious which code failed. An integration test, by its very nature, involves multiple components. When it fails, it might not be obvious which sub-component caused the error. If you find that you have integration tests failing and it takes a while to figure out what went wrong, then that's a sign that you should bolster your test suite with more unit tests. Once an integration test fails and one or more unit tests fail, then you have the best of both worlds: you've been made aware that you've broken a use case (integration test), but you also know which precise behavior is no longer working as before (unit test). [Tools and Techniques] [Tests are Code] Test code is just as important as product code. Use all of the same techniques to improve code quality in test code as you would in product code. Clean coding, good variable names, avoid copy/paste coding—all of it applies just as much to tests. There are two main differences: * You don't need to document tests. * You don't have to write tests for tests. 😀 [Writing testable code] This is a big, big topic, of course. There are a few guidelines that make it easier to write tests—or to avoid having to write tests at all. As noted above, code that can be validated by the compiler (static analysis) doesn't need tests. E.g. you don't have to write a test for how your code behaves when passed a null parameter if you just forbid it. Likewise, you don't have to re-verify that types work as they should in statically typed languages. We can trust the compiler. Here are a handful of tips. * Prefer composition to inheritance * A functional programming style is very testable * An IOC Container is very helpful * Avoid nullable properties, results, and parameters * Avoid mutable data * Interfaces are much easier to fake or mock; use those wherever you can See the following articles for more ideas. * "C# Handbook - Chapter 4: Design" (2017) * "Questions to consider when designing APIs: Part I" (2014) * "Questions to consider when designing APIs: Part II" (2014) * "Why use an IOC? (hint: testing)" (2019) [Parameterized Tests] Investigate your testing library to learn how to write multiple tests without having to write a lot of code. In the MSTests framework, you can use DataRow to parameterize a test. In NUnit, TestCase does the same thing, and Value allows you to provide parameter values for a list of tests that are the Cartesian product of all values. [Mocking/Faking] Use mocks or fakes to exclude a subsystem from a test. What would you want to exclude? While you will want to make some tests that include database access or REST API calls, there are a lot of tests where you're proving a fact that doesn't depend on these results. [Focus on what you're testing] For example, suppose a component reads its configuration from the database by default. A test of that component may simply want to see how it reacts with a given input to a given method. Where the configuration came from is irrelevant to that particular test. In that case, you could mock away the component that loads the configuration from the database and instead use a fake object that just provides some standard values. [Test error conditions] Another possibility is to fake an external service to see how your code reacts when the service returns an error or an ambiguous response. Without mocks, how would you test how your code reacts when a REST endpoint returns 503 or 404? Without a mock, how would you force the purely external endpoint to give a certain code? You really can't. With a mock, though, you can replace the service and return a 404 response for a specific test. This is quite a powerful technique. [How to fake?] As noted above, it's much, much easier to use fake objects if you've consistently used interfaces. You can just create your own implementation of the interface whose standard implementation you want to replace, give it a fake implementation (e.g. returning false and empty string and null for methods and properties), and then use that class as the implementation. [Faking/mocking libraries] If you have interfaces that perform a single task (single-responsibility principle), then it doesn't take too much effort to write the fake object by hand. However, it's much easier to use a library to create fake objects—and there are other benefits as well, like tracking which methods were called with which parameters. You can assert on this data collected by the fake object. For .NET, a great library for faking objects is "FakeItEasy" . With a fake object, you can indicate which values to return for a given set of parameters without too much effort. Similarly, you can use the same API to query how often these methods have been called. This allows you to verify, for example, that a call to a REST service would have been made. This is a powerful way of proving facts about your code without having to actually interact with external services. [An example] The following code configures a fake object for ITestUnitConfigurationService that returns default data for all properties, except for Configuration and GetTestUnitParameterValues(), which are configured to return specific data. private static ITestUnitConfigurationService CreateFakeTestUnitConfigurationService() { var result = A.Fake(); var testUnitParameters = CreateTestUnitParameters(); var testUnitConfiguration = new TestUnitConfiguration(testUnitParameters); A.CallTo(() => result.Configuration).Returns(testUnitConfiguration); var testUnitParameterValues = CreateTestUnitParameterValues(); A.CallTo(() => result.GetTestUnitParameterValues()).Returns(testUnitParameterValues); return result; } In the test, we could get this fake object back out of the IOC (for example) and then verify that certain methods have been called the expected number of times. var testUnitConfigurationService = locator.GetInstance(); A.CallTo(() => testUnitConfigurationService.Configuration).MustHaveHappenedOnceExactly(); A.CallTo(() => testUnitConfigurationService.GetTestUnitParameterValues()).MustHaveHappenedOnceExactly(); [Snapshot-testing] You can avoid writing a ton of assertions and a ton of tests with snapshot testing. For example, imagine you have a test that generates a particular view model. You want to verify 30 different parts of this complex model. You could navigate the data structure, asserting the 30 values individually. That would be pretty tedious, though, and lead to fragile and hard-to-maintain testing code. Instead, you could emit that structure as text and save it as a snapshot in the repository. If a future code change leads to a different snapshot, the test fails and the developer that caused the failure would have to approve the new snapshot (if it's an expected or innocuous change) or fix the code (if it was inadvertent and wrong). The upside is that large swaths of assertions are reduced to a simple snapshot assertion. The downside is that the test might break more often for spurious reasons. Generally, you can avoid these spurious reasons by being judicious about how your format the snapshot, * Avoid timestamps or data that changes over time * Avoid using output methods that are too likely to change over time See the documentation for the "Snapshooter NuGet package" . ]]> https://www.earthli.com/news/view_article.php?id=3666 2019-01-22T19:47:58+01:00 https://earthli.com/users/marco

https://www.earthli.com/news/view_article.php?id=3670 2019-01-21T20:26:52+01:00 https://earthli.com/users/marco

. According to the VS support team in that link, you have to install the "Visual Studio extension development" workload in the Visual Studio installer. In this package, the "Architecture and analysis tools" feature is available, but not included by default. Hovering this feature shows a tooltip indicating that it contains "Code Map, Live Dependency Validation and Code Clone detection". The "Live Dependency Validation" sounds like it might do what I want, but it also sounds quite heavyweight and somewhat intrusive, as described in "this blog from the end of 2016" . Instead of further modifying my VS installation (and possibly slowing it down), I decided to try another tool. [ReSharper?] What about ReSharper? For a while now, it's included project-dependency graphs and hierarchies. Try as I might, I couldn't get the tools to show me the transitive dependency on PostgreSql that Sandbox Winform was pulling in from somewhere. The hierarchy view is live and quick, but it doesn't show all transitive usages. [image] The graph view is nicely rendered, but shows dependencies by default instead of dependencies and usages. At any rate, the Sandbox wasn't showing up as a transitive user of PostgreSql. [image] I didn't believe ReSharper at this point because something was causing the data driver to be copied to the output folder. [NDepend to the rescue ] So, as expected, I turned to NDepend. I took a few seconds to run an analysis and then right-clicked the PostgreSql data-driver project to select NDepend => Select Assemblies... => That are Using Me (Directly or Indirectly) to show the following query and results. [image] Bingo. Sandbox.Model is indirectly referencing the PostgreSql data driver, via a transitive-dependency chain of 4 assemblies. Can I see which assemblies they are? Of course I can: this kind of information is best shown on a graph, so you can show a graph of any query results by clicking Export to Graph to show the graph below. [image] Now I can finally see that the SandboxModel pulls in the Quino.Testing.Models.Generated (to use the BaseTypes module) which, in turn, has a reference to Quino.Tests.Base which, of course, includes the PostgreSql driver because that's the default testing driver for Quino tests. Now that I know how the reference is coming in, I can fix the problem. Here I'm on my own: I have to solve this problem without NDepend. But at least NDepend was able to show me exactly what I have to fix (unlike VS or ReSharper). I ended up moving the test-fixture base classes from Quino.Testing.Models.Generated into a new assembly called Quino.Testing.Models.Fixtures. The latter assembly still depends on Quino.Tests.Base and thus the PostgreSql data driver, but it's now possible to reference the Quino testing models without transitively referencing the PostgreSql data driver. A quick re-analysis with NDepend and I can see that the same query now shows a clean view: only testing code and testing assemblies reference the PostgreSql driver. [image] [Finishing up] And now to finish my original task! I ran the Winform Sandbox application with the PostgreSql driver configured and was greeted with an error message that the driver could not be loaded. I now had parity between PostgreSql and SqlServer. The fix? Obviously, make sure that the drivers are available by referencing them directly from any Sandbox application that needs to connect to a database. This was the obvious solution from the beginning, but we had to quickly fix a problem with dependencies first. Why? Because we hate hacking. :-) Two quick references added, one build and I was able to connect to both SQL Server and PostgreSql. ]]> https://www.earthli.com/news/view_article.php?id=3681 2019-01-20T22:37:35+01:00 https://earthli.com/users/marco

or "here at earthli" . The following excerpts should give you an idea of what you're in for, should you download and read the 80-page document. [Details] The TOC lists the following top-level chapters: 1. Introduction 2. Examples 3. Context & Scopes 4. Standard Queries 5. Grouping Queries 6. Evaluation 7. Syntax 8. Data Types and Operators 9. Libraries 10. Best Practices 11. Implementation Details 12. Future Enhancements From the abstract in the document: "The Quino Query Language (QQL) defines a syntax and semantics for formulating data requests against hierarchical data structures. It is easy to read and learn both for those familiar with SQL and non-programmers with a certain capacity for abstract thinking (i.e. power users). Learning only a few basic rules is enough to allow a user to quickly determine which data will be returned by all but the more complex queries. As with any other language, more complex concepts result in more complex texts, but the syntax of QQL limits these cases." From the overview: "QQL defines a syntax and semantics for writing queries against hierarchical data structures. A query describes a set of data by choosing an initial context in the data and specifying which data are to be returned and how the results are to be organized. An execution engine generates this result by applying the query to the data." [Examples] [Standard Projections] The follow is from chapter 2.1, "Simple Standard Query": The following query returns the first and last name of all active people as well as their 10 most recent time entries, reverse-sorted first by last name, then by first name. Person { select { FirstName; LastName; Sample:= TimeEntries { orderby Date desc; limit 10 } } where Active orderby { LastName desc; FirstName desc; } } In chapter 2, there are also "2.2 Intermediate Standard Query" and "2.3 Complex Standard Query" examples. [Grouping Projections] The following is from chapter 2.4, "Simple Grouping Query": The following query groups active people by last name and returns the age of the youngest person and the maximum contracts for each last name. Results are ordered by the maximum contracts for each group and then by last name. group Person { groupby LastName; select { default; Age:= (Now - BirthDate.Min).Year; MaxContracts:= Contracts.Count.Max } where Active; orderby { MaxContracts desc; LastName desc; } } In chapter 2, there are also "2.5 Complex Grouping Query", "2.6 Standard Query with Grouping Query" and "2.7 Nested Grouping Queries" examples. ]]> https://www.earthli.com/news/view_article.php?id=3682 2019-01-20T22:19:02+01:00 https://earthli.com/users/marco

( out T value, TKey key = default(TKey), [CanBeNull] ILogger logger = null ); We would like to remove the logger parameter. So we deprecate the method above and declare the new method. bool TryGetValue( out T value, TKey key = default(TKey) ); Now the compiler/ReSharper notifies you that there will be an ambiguity if a caller does not pass a logger. How to resolve this? Well, we can just remove the default value for that parameter in the obsolete method. bool TryGetValue( out T value, TKey key = default(TKey), [CanBeNull] ILogger logger ); But now you've got another problem: The parameter logger cannot come after the key parameter because it doesn't have a default value. So, now you'd have to move the logger parameter in front of the key parameter. This will cause a compile error in clients, which is what we were trying to avoid in the first place. In this case, we have a couple of sub-optimal options. Multiple Releases Use a different name for the new API (e.g. TryGetValueEx à la Windows) in the next major version, then switch the name back in the version after that and finally remove the obsolete member in yet another version. That is, * in version n, TryGetValue (with logger) is obsolete and users are told to use TryGetValueEx (no logger) * in version n+1, TryGetValueEx (no logger) is obsolete and users are told to use TryGetValue (no logger) * in version n+2, we finally remove TryGetValueEx. This is a lot of work and requires three upgrades to accomplish. You really need to stay on the ball in order to get this kind of change integrated and it takes a non-trivial amount of time and effort. We generally don't use this method, as our customers are developers and can deal with a compile error or two, especially when it's noted in the release notes and the workaround is fairly obvious (e.g. the logger parameter is just no longer required). Remove instead of deprecating Accept that there will be a compile error and soften the landing as much as possible for customers by noting it in the release notes. ]]> https://www.earthli.com/news/view_article.php?id=3680 2019-01-20T22:00:30+01:00 https://earthli.com/users/marco

for more information. [Examples] * 0.9.0-alpha34: A pre-release of 0.9.0 * 0.9.0-beta48: A pre-release of 0.9.0 * 0.9.0.67: An official release of 0.9.0 * 1.0.0-rc512: A pre-release of 1.0.0 * 1.0.0.523: An official release of 1.0.0 The numbers are strictly ordered. The first three *parts* indicate the "main" version. The final *part* counts strictly upward. [Parts] The following list describes each of the parts and explains what to expect when it changes. [Build] * Identifies the build task that produced the artifact * Strictly increasing [Label] * An arbitrary designation for the "type" of pre-release [Patch] * Introduces bug fixes but no features or API changes * May introduce obsolete members * May *not* introduce breaking changes This part is also known as "Maintenance" (see "versioning">Software versioning" on Wikipedia). [Minor] * Introduces new features that extend existing functionality * May include bug fixes * May cause minor breaking changes * May introduce obsolete members that cause compile errors * Workarounds must be documented in release notes or obsolete messages [Major] * Introduces major new features * Introduces breaking changes that require considerable effort to integrate * Introduces a new data or protocol format that requires migration [Conventions] [Uniqueness for official releases] There will only ever be one artifact of an official release corresponding to a given "main" version number. That is, if 1.0.0.523 exists, then there will never be a 1.0.0.524. This is due the fact that the build number (e.g. 524) is purely for auditing. For example, suppose your software uses a NuGet package with version 1.0.0.523. NuGet will not offer to upgrade to 1.0.0.524. [Pre-release Labels] There are no restrictions on the labels for pre-releases. However, it's recommended to use one of the following: * alpha * beta * rc Be aware that if you choose a different label, then it is ordered alphabetically relative to the other pre-releases. For example, if you were to use the label pre-release to produce the version 0.9.0-prealpha21, then that version is considered to be higher than 0.0.0-alpha34. A tool like NuGet will not see the latter version as an upgrade. [Release branches] The name of a release branch should be the major version of that release. E.g. release/1 for version 1.x.x.x. [Pre-release branches] The name of a pre-release branch should be of the form feature/[label] where [label] is one of the labels recommended above. It's also OK to use a personal branch to create a pre-release build, as in mvb/[label]. [Setting the base version] A developer uses the quino tool to set the version. For example, to set the version to 1.0.1, execute the following: quino fix -v 1.0.1.0 The tool will have updated the version number in all relevant files. [Calculating final version] The build server calculates a release's version number as follows, * major: Taken from solution * minor: Taken from solution * maintenance: Read from solution * label: Taken from the Git branch (see below for details) * build: Provided by the build server [Git Branches] The name of the Git branch determines which kind of release to produce. * If the name of the branch matches the glob **/release/*, then it's an official release * Everything else is a pre-release For example, * origin/release/1 * origin/production/release/new * origin/release/ * release/1 * production/release/new * release/ The name of the branch doesn't influence the version number since an official release doesn't have a label. [Pre-release labels] The label is taken from the last part of the branch name. For example, * origin/feature/beta yields beta * origin/feature/rc yields rc * origin/mvb/rc yields rc The following algorithm ensures that the label can be part of a valid semantic version. * Remove invalid characters * Append an X after a trailing digit * Use X if the label is empty (or becomes empty after having removed invalid characters) For example, * origin/feature/rc1 yields rc1X * origin/feature/linuxcompat yields linuxcompat * origin/feature/12 yields X [Examples] Assume that, * the version number in the solution is 0.9.0.0 * the build counter on the build server is at 522 Then, * Deploying from branch origin/release/1 produces artifacts with version number 0.9.0.522 * Deploying from branch origin/feature/rc produces artifacts with version number 0.9.0-rc522 [Release Workflow] The following are very concise guides for how to produce artifacts. [Pre-release] * Ensure you are on a non-release branch (e.g. feature/rc, master) * Verify or set the base version (e.g. quino fix -v 1.0.2.0 * Push any changes to Git * Execute the "deploy" task against your branch on the build server [Release] * Ensure you are on a release branch (e.g. release/1) * Verify or set the base version (e.g. quino fix -v 1.0.2.0`) * Push any changes to Git * Execute the "deploy" task against your branch on the build server ]]> https://www.earthli.com/news/view_article.php?id=3679 2019-01-20T21:59:55+01:00 https://earthli.com/users/marco

at the end of October, 2018. ------------------------------------------------------------------------ The summary below describes major new features, items of note and breaking changes. * "Download the artifacts" * See the full list of issues * "Quino-Standard 6.0.0" * "Quino-WebAPI 6.0.0" * "Quino-Windows 6.0.0" The... ]]>

at the end of October, 2018. -------------------------------------------------------------------------------- The summary below describes major new features, items of note and breaking changes. * "Download the artifacts" * See the full list of issues * "Quino-Standard 6.0.0" * "Quino-WebAPI 6.0.0" * "Quino-Windows 6.0.0" The links above require a login. [Overview] At long last, Quino enters the world of .NET Standard and .NET Core. Libraries target .NET Standard 2.0, which means they can all be used with any .NET runtime on any .NET platform (e.g. Mac and Linux). Sample applications and testing assemblies target .NET Core 2.0. Tools like quinogenerate and quinofix target .NET Core 2.1 to take advantage of the standardized external tool-support there. Furthermore, the Windows, Winform and WPF projects have moved to a separate solution/repository called Quino-Windows. Quino-Standard is the core on which both Quino-Windows and Quino-WebAPI build. * All core assemblies target .NET Standard 2.0. * All assemblies in Quino-Windows target .NET Framework 4.6.2 because that's the first framework that can interact with .NET Standard (and under which Windows-specific code runs). * All assemblies in Quino-WebAPI currently target .NET Framework 4.6.2. We plan on targeting .NET Core in an upcoming version (tentatively planned for v7). [Highlights] * Target .NET Standard and .NET Core from Quino-Standard * Split Windows-specific code to Quino-Windows * Improve authentication API to use IIdentity everywhere (deprecating ICredentials and IUserCredentials). [Breaking Changes] 6.0 is a pretty major break from the 5.x release. Although almost all assembly names have stayed the same, we had to move some types around to accommodate targeting .NET Standard with 85% of Quino's code. [APIs] We've tried to support existing code wherever possible, but some compile errors will be unavoidable (e.g. from namespace changes or missing references). In many cases, R#/VS should be able to help repair these errors. These are the breaking changes that are currently known. * Moved IRunSettings and RunMode from Encodo.Application to Encodo.Core. [References] Any .NET Framework executable that uses assemblies targeting .NET Standard must reference .NET Standard itself. The compiler (MSBuild) in Visual Studio will alert you to add a reference to .NET Standard using NuGet. This applies not just to Winform executables, but also to any unit-test assemblies. [Tools] One piece that has changed significantly is the tool support formerly provided with Quino.Utils. As of version 6, Quino no longer uses NAnt, instead providing dotnet-compatible tools that you can install using common .NET commands. Currently, Quino supports: * dotnet quinofix * dotnet quinogenerate * dotnet quinopack Please see the "tools documentation" for more information on how to install and use the new tools. The standalone Winforms-based tools are in the Quino-Windows download, in the Tools.zip archive. * Quino.Migrator * Quino.PasswordEncryptor Quino.Utils is no longer supported as a NuGet package. ]]> https://www.earthli.com/news/view_article.php?id=3673 2019-01-20T21:59:29+01:00 https://earthli.com/users/marco

in July, 2018. ------------------------------------------------------------------------ In recent articles, we outlined "a roadmap to .NET Standard and .NET Core" and "a roadmap for deployment and debugging" . These two roadmaps taken together illustrate our plans to extend as much of Quino as possible to other... ]]>

in July, 2018. -------------------------------------------------------------------------------- In recent articles, we outlined "a roadmap to .NET Standard and .NET Core" and "a roadmap for deployment and debugging" . These two roadmaps taken together illustrate our plans to extend as much of Quino as possible to other platforms (.NET Standard/Core) and to make development with Quino as convenient as possible (getting/upgrading/debugging). To round it off, we've made good progress on another vital piece of any framework: documentation. [Introducing docs.encodo.ch] We recently set up a new server to host "Quino documentation" . There, you can find documentation for current releases. Going forward, we'll also retain documentation for any past releases. We're generating our documentation with "DocFX" , which is the same system that powers Microsoft's own documentation web site. We've integrated documentation-generation as a build step in Quino's nightly build on TeamCity, so it's updated every night (Zürich time) [1]. The documentation includes "conceptual documentation" which provides an overview/tutorials/FAQ for basic concepts in Quino. The "API Reference" includes comprehensive documentation about the types and methods available in Quino. [Next Steps] While we're happy to announce that we have publicly available documentation for Quino, we're aware that we've got work to do. The next steps are: * Improve and extend conceptual documentation ("QNO-3452" , "QNO-3453" , "QNO-3985" , "QNO-5282" , "QNO-5283" , "QNO-5284" , "QNO-5286" , "QNO-5391" , "QNO-5528" , "QNO-5544" , "QNO-5562" , "QNO-5813" , "QNO-5912" ) * Improve the search for API documentation to include members as well as types ("QNO-5934" ) * Integrate documentation for the newly created/separated Quino-WebApi repository ("IT-1293" ) * Retain documentation for the soon-to-be-split-off Quino-Windows repository ("QNO-5904" ) Even though there's still work to do, this is a big step in the right direction. We're very happy to have found DocFX, which is a very comprehensive, fast and nice-looking solution to generating documentation for .NET code. [2] -------------------------------------------------------------------------------- [1] If the build succeeds, naturally. :-) [1] We used to use Sandcastle many years ago, but dropped support because it took forever to generate documentation, required its own solution file, didn't look very nice out-of-the-box, wasn't so easily customized and didn't have a very good search (which also didn't work without an IIS running it). ]]> https://www.earthli.com/news/view_article.php?id=3672 2019-01-20T21:58:53+01:00 https://earthli.com/users/marco

in July, 2018. ------------------------------------------------------------------------ In a recent article, we outlined "a roadmap to .NET Standard and .NET Core" . We've made really good progress on that front: we have a branch of Quino-Standard that targets .NET Standard for class libraries and .NET Core for... ]]>

in July, 2018. -------------------------------------------------------------------------------- In a recent article, we outlined "a roadmap to .NET Standard and .NET Core" . We've made really good progress on that front: we have a branch of Quino-Standard that targets .NET Standard for class libraries and .NET Core for utilities and tests. So far, we've smoke-tested these packages with Quino-WebApi. Our next steps there are to convert Quino-WebApi to .NET Standard and .NET Core as well. We'll let you know when it's ready, but progress is steady and promising. With so much progress on several fronts, we want to address how we get Quino from our servers to our customers and users. [Getting Quino] Currently, we provide access to a private fileshare for customers. They download the NuGet packages for the release they want. They copy these to a local folder and bind it as a NuGet source for their installations. In order to make a build available to customers, we have to publish that build by deploying it and copying the files to our file share. This process has been streamlined considerably so that it really just involves telling our CI server (TeamCity) to deploy a new release (official or pre-). From there, we download the ZIP and copy it to the fileshare. Encodo developers don't have to use the fileshare because we can pull packages directly from TeamCity as soon as they're available. This is a much more comfortable experience and feels much more like working with nuget.org directly. [Debugging Quino] The debugging story with external code in .NET is much better than it used to be (spoiler: it was almost impossible, even with Microsoft sources), but it's not as smooth as it should be. This is mostly because NuGet started out as a packaging mechanism for binary dependencies published by vendors with proprietary/commerical products. It's only in recent year(s) that packages are predominantly open-source. In fact, debugging with third-party sources -- even without NuGet involved -- has never been easy with .NET/Visual Studio. Currently, all Quino developers must download the sources separately (also available from TeamCity or the file-share) in order to use source-level debugging. Binding these sources to the debugger is relatively straightforward but cumbersome. Binding these sources to ReSharper is even more cumbersome and somewhat unreliable, to boot. I've created the issue "Add an option to let the user search for external sources explicitly (as with the VS debugger) when navigating" in the hopes that this will improve in a future version. JetBrains has already fixed one of my issues in this are ("Navigate to interface/enum/non-method symbol in Nuget-package assembly does not use external sources" ), so I'm hopeful that they'll appreciate this suggestion, as well. The use case I cited in the issue above is, Developers using NuGet packages that include sources or for which sources are available want to set breakpoints in third-party source code. Ideally, a developer would be able to use R# to navigate through these sources (e.g. via F12) to drill down into the code and set a breakpoint that will actually be triggered in the debugger. As it is, navigation in these sources is so spotty that you often end up in decompiled code and are forced to use the file-explorer in Windows to find the file and then drag/drop it to Visual Studio where you can set a breakpoint that will work. The gist of the solution I propose is to have R# ask the user where missing sources are before decompiling (as the Visual Studio debugger does). [Nuget Protocol v3 to the rescue?] There is hope on the horizon, though: Nuget is going to address the debugging/symbols/sources workflow in an upcoming release. The overview is at "NuGet Package Debugging & Symbols Improvements" and the issue is "Improve NuGet package debugging and symbols experience" . Once this feature lands, Visual Studio will offer seamless support for debugging packages hosted on nuget.org. Since we're using TeamCity to host our packages, we need JetBrains to "Add support for NuGet Server API v3" [1] in order to benefit from the improved experience. Currently, our customers are out of luck even if JetBrains releases simultaneously (because our TeamCity is not available publicly). [Quino goes public?] I've created an issue for Quino, "Make Quino Nuget packages available publicly" to track our progress in providing Quino packages to our customers in a more convenient way that also benefits from improvements to the debugging workflow with Nuget Packages. If we published Quino packages to NuGet (or MyGet, which allows private packages), then we would have the benefit of the latest Nuget protocol/improvements for both ourselves and our customers as soon as it's available. Alternatively, we could also proxy our TeamCity feed publicly. We're still considering our options there. As you can see, we're always thinking about the development experience for both our developers and our customers. We're fine-tuning on several fronts to make developing and debugging with Quino a seamless experience for all developers on all platforms. We'll keep you posted. -------------------------------------------------------------------------------- [1] This issue has been closed and released with R# in the latest versions. ]]> https://www.earthli.com/news/view_article.php?id=3674 2019-01-20T21:55:36+01:00 https://earthli.com/users/marco

in July, 2018. ------------------------------------------------------------------------ The title is a bit specific for this blog post, but that's the gist of it: we ended up with a bunch of references to an in-between version of .NET (4.6.1) that was falsely advertising itself as a more optimal candidate for... ]]>

in July, 2018. -------------------------------------------------------------------------------- The title is a bit specific for this blog post, but that's the gist of it: we ended up with a bunch of references to an in-between version of .NET (4.6.1) that was falsely advertising itself as a more optimal candidate for satisfying 4.6.2 dependencies. This is a known issue; there are several links to MS GitHub issues below. In this blog, I will discuss direct vs. transient dependencies as well as internal vs. runtime dependencies. [tl;dr] If you've run into problems with an application targeted to .NET Framework 4.6.2 that does not compile on certain machines, it's possible that the binding redirects Visual Studio has generated for you use versions of assemblies that aren't installed anywhere but on a machine with Visual Studio installed. How I solved this issue: * Remove the C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461\ directory * Remove all System* binding redirects * Clean out all bin/ and obj/ folders * Delete the .vs folder (may not be strictly necessary) * Build in Visual Studio * Observe that a few binding-redirect warnings appear * Double-click them to re-add the binding redirects, but this time to actual 4.6.2 versions (you may need to add true to your project) * Rebuild and verify that you have no more warnings The product should now run locally and on other machines. For more details, background and the story of how I ran into and solved this problem, read on. Note: I published a recent article, ".NET Tips and Resources" , containing a link to a video by Immo Landwerth, in which says "If you want to be compatible with .NET Core 1.5 or lower, then you can use .NET Framework 4.6.1. For .NET Standard compatibility, you should definitely use .NET Framework 4.7.2 instead." That will probably fix the problem as well. Moving to .NET Core will also fix the problem, as all binding is handled automatically there. [Building Software] What do we mean when we say that we "build" an application? Building is the process of taking a set of inputs and producing an artifact targeted at a certain runtime. Some of these inputs are included directly while others are linked externally. * Examples of direct inputs are the binary artifacts produced from the source code that comprises your application * Examples of external inputs are OS components and runtime environments The machine does exactly what you tell it to, so it's up to you to make sure that your instructions are as precise as possible. However, you also want your application to be flexible so that it can run on as wide an array of environments as possible. Your source code consists of declarations. We've generally got the direct inputs under control. The code compiles and produces artifacts as expected. It's the external-input declarations where things go awry. What kind of external inputs does our application have? * System dependencies in the runtime target (assemblies like System.Runtime, System.Data, etc.), each with a minimum version * Third-party dependencies pulled via NuGet, each with a minimum version How is this stitched together to produce the application that is executed? * The output folder contains our application, our own libraries and the assemblies from NuGet dependencies * All other dependencies (e.g. system dependencies) are pulled from the environment The NuGet dependencies are resolved at build time. All resources are pulled and added to the release on the build machine. There are no run-time decisions to make about which versions of which assemblies to use. Dependencies come in two flavors: * Direct: A reference in the project itself * Transient: A direct reference inherited from another direct or transient reference It is with the transient references that we run into issues. The following situations can occur: * A transient dependency is referenced one or more times with the same version. This is no problem, as the builder simply uses that version or substitutes a newer version if that version is no longer available (rare, but possible) * A transient dependency is referenced in different versions. In this case, the builder tries to substitute a single version for all requirements. This generally works OK since most dependencies require a given version or higher. It may be that one or another library cannot work with all newer versions, but this is also rare. In this case, the top-level assembly (the application) must include a hint (an assembly-binding redirect) that indicates that the substitution is OK. More on these below. * A transient dependency requires a lower version than the version that is directly referenced. This is also not a problem, as the transient dependency is satisfied by the direct dependency with the higher version. In this case, the top-level application must also include an assembly-binding redirect to allow the substitution without warning. * A transient dependency requires a higher version than the version that is directly referenced. This is an error (no longer just a warning) that must be solved by either downgrading the dependency that leads to the problematic transient dependency or upgrading the direct dependency. Generally, the application will upgrade the direct dependency. [Assembly-Binding Redirects] An application generally includes an app.config (desktop applications or services) or web.config XML file that includes a section where binding redirects are listed. A binding redirect indicates the range of versions that can be mapped (or redirected) to a certain fixed version (which is generally also included as a direct dependency). A redirect looks like this (a more-complete form is further below): When the direct dependency is updated, the binding redirect must be updated as well (generally by updating the maximum version number in the range and the version number of the target of the redirect). NuGet does this for you when you're using package.config. If you're using Package References, you must update these manually. This situation is currently not so good, as it increases the likelihood that your binding redirects remain too restrictive. [NuGet Packages] NuGet packages are resolved at build time. These dependencies are delivered as part of the deployment. If they could be resolved on the build machine, then they are unlikely to cause issues on the deployment machine. [System Dependencies] Where the trouble comes in is with dependencies that are resolved at execution time rather than build time. The .NET Framework assemblies are resolved in this manner. That is, an application that targets .NET Framework expects certain versions of certain assemblies to be available on the deployment machine. We mentioned above that the algorithm sometimes chooses the desired version or higher. This is not the case for dependencies that are in the assembly-binding redirects. Adding an explicit redirect locks the version that can be used. This is generally a good idea as it increases the likelihood that the application will only run in a deployment environment that is extremely close or identical to the development, building or testing environment. [Aside: Other Bundling Strategies] How can we avoid these pesky run-time dependencies? There are several ways that people have come up with, in increasing order of flexibility: * Deliver hardware and software together. This is common in industrial applications and used to be much more common for businesses, as well. Nearly bulletproof. If it worked in the factory, it will work for the customer. * Deliver a VM (virtual machine) as your application. This includes the entire execution environment right down to the hardware. Safe, but inefficient. * Use a container (e.g. Docker) to deliver a description of the execution environment. The image is built to match the declaration. This is also quite stable and can avoid many of the substitution errors outlined above. If components are outdated, the machine fails to start and the definition must first be updated (and, presumably, tested). This type of deployment is getting more reliable but is also overkill for many applications. * Deliver the runtime with the application instead of describing the runtime you'd like to have. Targeting .NET Core instead of .NET Framework includes the runtime. This seems like a nice alternative and it's not surprising that Microsoft went in this direction with .NET Core. It's a good solution to the external-dependency issues outlined above. To sum up: * A VM delivers the OS, runtime and application. * A Container delivers a description of the OS and runtime as well as the application itself. * .NET Core includes the runtime and application and is OS-agnostic (within reason). * .NET Framework includes only the application and some directives on the remaining components to obtain from the runtime environment. Our application targets .NET Framework (for now). We're looking into .NET Core, but aren't ready to take that step yet. [Where can the deployment go wrong?] To sum up the information from above, problems arise when the build machine contains components that are not available on the deployment machine. How can this happen? Won't the deployment machine just use the best match for the directives included in the build? Ordinarily, it would. However, if you remember our discussion of assembly-binding redirects above, those are set in stone. What if you included binding redirects that required versions of system dependencies that are only available on your build machine ... or even your developer machine? [Special Tip for Web Applications] We actually discovered an issue in our deployment because the API server was running, but the Authentication server was not. The Authentication server was crashing because it couldn't find the runtime it needed in order to compile its Razor views (it has ASP.Net MVC components). We only discovered this issue on the deployment server because the views were only ever compiled on-the-fly. To catch these errors earlier in the deployment process, you can enable pre-compiling views in release mode so that the build server will fail to compile instead of a producing a build that will sometimes fail to run. Add the true to any MVC projects in the PropertyGroup for the release build, as shown in the example below: pdbonly true bin TRACE prompt 4 6 true [How do I create a redirect?] We mentioned above that NuGet is capable of updating these redirects when the target version changes. An example is shown below. As you can see, they're not very easy to write: Most bindings are created automatically when MSBuild emits a warning that one would be required in order to avoid potential runtime errors. If you compile with MSBuild in Visual Studio, the warning indicates that you can double-click the warning to automatically generate a binding. If the warning doesn't indicate this, then it will tell you that you should add the following to your project file: true After that, you can rebuild to show the new warning, double-click it and generate your assembly-binding redirect. [How did we get the wrong redirects?] When MSBuild generates a redirect, it uses the highest version of the dependency that it found on the build machine. In most cases, this will be the developer machine. A developer machine tends to have more versions of the runtime targets installed than either the build or the deployment machine. A Visual Studio installation, in particular, includes myriad runtime targets, including many that you're not using or targeting. These are available to MSBuild but are ordinarily ignored in favor of more appropriate ones. That is, unless there's a bit of a bug in one or more of the assemblies included with one of the SDKs...as there is with the net461 distribution in Visual Studio 2017. Even if you are targeting .NET Framework 4.6.2, MSBuild will still sometimes reference assemblies from the 461 distribution because the assemblies are incorrectly marked as having a higher version than those in 4.6.2 and are taken first. I found the following resources somewhat useful in explaining the problem (though none really offer a solution): * ".dll in Microsoft.NET.Build.Extensions\net461 targets .NET Framework 4.6.1" * "ImplicitlyExpandNETStandardFacades option on MSBuild Extensions corrupts build" (GitHub issue) * "MS Build Extensions file corrupt my bin (Web API) folder" How can you fix the problem if you're affected? You'll generally have a crash on the deployment server that indicates a certain assembly could not be loaded (e.g. System.Runtime). If you show the properties for that reference in your web application, do you see the path C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461 somewhere in there? If so, then your build machine is linking in references to this incorrect version. If you let MSBuild generate binding redirects with those referenced paths, they will refer to versions of runtime components that do not generally exist on a deployment machine. Tips for cleaning up: * Use MSBuild to debug this problem. R# Build is nice, but not as good as MSBuild for this task. * Clean and Rebuild to force all warnings * Check your output carefully. * Do you see warnings related to package conflicts? * Ambiguities? * Do you see the path C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461 in the output? A sample warning message: [ResolvePackageFileConflicts] Encountered conflict between Platform:System.Collections.dll and CopyLocal:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461\lib\System.Collections.dll. Choosing CopyLocal:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461\lib\System.Collections.dll because AssemblyVersion 4.0.11.0 is greater than 4.0.10.0. [The Solution] As mentioned above, but reiterated here, this what I did to finally stabilize my applications: * Remove the C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461\ directory * Remove all System* binding redirects * Clean out all bin/ and obj/ folders * Delete the .vs folder (may not be strictly necessary) * Build in Visual Studio * Observe that a few binding-redirect warnings appear * Double-click them to re-add the binding redirects, but this time to actual 4.6.2 versions (you may need to add true to your project) * Rebuild and verify that you have no more warnings * Deploy and TADA! [One more thing] When you install any update of Visual Studio, it will silently repair these missing files for you. So be aware and check the folder after any installations or upgrades to make sure that the problem doesn't creep up on you again. ]]> https://www.earthli.com/news/view_article.php?id=3671 2019-01-20T21:49:13+01:00 https://earthli.com/users/marco

in May, 2018. ------------------------------------------------------------------------ With Quino 5, we've gotten to a pretty good place organizationally. Dependencies are well-separated into projects—and there are almost 150 of them. We can use code-coverage, solution-wide-analysis and so on without a... ]]>

in May, 2018. -------------------------------------------------------------------------------- With Quino 5, we've gotten to a pretty good place organizationally. Dependencies are well-separated into projects—and there are almost 150 of them. We can use code-coverage, solution-wide-analysis and so on without a problem. TeamCity runs the ~10,000 tests quickly enough to provide feedback in a reasonable time. The tests run even more quickly on our desktops. It's a pretty comfortable and efficient experience, overall. [Monolithic Solution: Pros and Cons] As of Quino 5, all Quino-related code was still in one repository and included in a single solution file. Luckily for us, Visual Studio 2017 (and Rider and Visual Studio for Mac) were able to keep up quite well with such a large solution. Recent improvements to performance kept the experience quite comfortable on a reasonably equipped developer machine. Having everything in one place is both an advantage and disadvantage: when we make adjustments to low-level shared code, the refactoring is applied in all dependent components, automatically. If it's not 100% automatic, at least we know where we need to make changes in dependent components. This provides immediate feedback on any API changes, letting us fine-tune and adjust until the API is appropriate for known use cases. On the other hand, having everything in one place means that you must make sure that your API not only works for but compiles and tests against components that you may not immediately be interested in. For example, we've been pushing much harder on the web front lately. Changes we make in the web components (or in the underlying Quino core) must also work immediately for dependent Winform and WPF components. Otherwise, the solution doesn't compile and tests fail. While this setup had its benefits, the drawbacks were becoming more painful. We wanted to be able to work on one platform without worrying about all of the others. On top of that, all code in one place is no longer possible with cross-platform support. Some code—Winform and WPF—doesn't run on Mac or Linux. [1] The time had come to separate Quino into a few larger repositories. [Separate Solutions] We decided to split along platform-specific lines. * Quino-Standard: all common code, including base libraries, application, configuration and IOC support, metadata, builders and all data drivers * Quino-WebApi: all web-related code, including remaining ASP.NET MVC support * Quino-Windows: all Windows-platform-only code (Windows-only APIs (i.e. native code) as well as Winform and WPF) The Quino-WebApi and Quino-Windows solution will consume Quino-Standard via NuGet packages, just like any other Quino-based product. And, just like any Quino-based product, they will be able to choose when to upgrade to a newer version of Quino-Standard. [Quino-Standard] Part of the motivation for the split is cross-platform support. The goal is to target all assemblies in Quino-Standard to .NET Standard 2.0. The large core of Quino will be available on all platforms supported by .NET Core 2.0 and higher. This work is quite far along and we expect to complete it by August 2018. [Quino-WebApi] As of Quino 5.0.5, we've moved web-based code to its own repository and set up a parallel deployment for it. Currently, the assemblies still target .NET Framework, but the goal here is to target class libraries to .NET Standard and to use .NET Core for all tests and sample web projects. We expect to complete this work by August 2018 as well. [Quino-Windows] We will be moving all Winform and WPF code to its own repository, setting it up with its own deployment (as we did with Quino-WebApi). These projects will remain targeted to .NET Framework 4.6.2 (the lowest version that supports interop with .NET Standard assemblies). We expect this work to be completed by July 2018. [Quino-Mobile] One goal we have with this change is to be able to use Quino code from Xamarin projects. Any support we build for mobile projects will proceed in a separate repository from the very beginning. We'll keep you posted on work and improvements and news in this area. [Conclusion] Customer will, for the most part, not notice this change, except in minor version numbers. Core and platform versions may (and almost certainly will) diverge between major versions. For major versions, we plan to ship all platforms with a single version number. -------------------------------------------------------------------------------- [1] I know, Winform can be made to run on Mac using Mono. And WPF may eventually become a target of Xamarin. But a large part of our Winform UI uses the Developer Express components, which aren't going to run on a Mac. And the plans for WPF on Mac/Linux are still quite up in the air right now. ]]> https://www.earthli.com/news/view_article.php?id=3683 2019-01-20T21:44:30+01:00 https://earthli.com/users/marco

(with examples). However, Encodo's blogging back-end now uses Umbraco, with Markdown for content. I used to be able to cross-post with ease, by copy/pasting. Now, I need to convert the content from Markdown to... ]]>

(with examples). However, Encodo's blogging back-end now uses Umbraco, with Markdown for content. I used to be able to cross-post with ease, by copy/pasting. Now, I need to convert the content from Markdown to earthli formatting. The following steps suffice to convert any article: 1. If there are attached media (e.g. graphics), save those locally 2. Create a new earthli article with the same title as the source article 3. Attach all media to the new article 4. Copy the main text; paste into Sublime Text 5. Find/replace the following regular expressions with the replacements: * ### ([^\n]+)$ => \1 * ## ([^\n]+)$ => \1 * \[([^!][^\]]+)\]\(([^\)]+)\) => \1 * \*\*([^\*]+)\*\* => \1 * _([^_]+)_ => \1 * ```txt\n([^`]+)\n``` =>

\1

* ```[a-z]+\n([^`]+)\n``` => \1 * `([^`]+)` => \1 6. Manually re-attach all media in the appropriate locations [1] I haven't automated this process yet because I only rarely transfer articles. -------------------------------------------------------------------------------- [1] This is easy since the earthli UI includes an attachment formatter that lets you place, align, link and scale the attachment better than in the original specification anyway. ]]> https://www.earthli.com/news/view_article.php?id=3554 2019-01-08T22:46:23+01:00 https://earthli.com/users/marco

https://www.earthli.com/news/view_article.php?id=3598 2019-01-08T22:28:18+01:00 https://earthli.com/users/marco

presents issues in JavaScript and a solution: use another language. The list several newer ones that are completely untested. But the main problem that the article mentions can't be solved 100% by any language. The... ]]>

presents issues in JavaScript and a solution: use another language. The list several newer ones that are completely untested. But the main problem that the article mentions can't be solved 100% by any language. The main problem is at the boundaries of your application: inputs. When you get data from an external source, you have to validate it somehow before passing it along to the rest of the application. No language can remove this requirement. It doesn't matter how functional, curryable, immutable or sexy it is; it just can't do it. What you have instead is languages with more built-in mechanisms for defining types that allow the rest of the program to work safely with the data, once it's been validated. So if your language supports immutability and types, then you can validate that the data is OK before hydrating the object from the serialized source (e.g. JSON). What we're trying to avoid is unexpected runtime errors, no? Or, at the very least, we want a runtime error of a known type that precisely identifies the problem with the incoming data. That is, the data either conforms to the definition -- and the definition is statically typed -- or there is an error. The desire is to push this gatekeeper/conversion to a single place so that the rest of the application works with the compiler to find errors rather than tyhe programmer defensively checking throughout the source. However, suggesting that PureScript or Elm or ClojureScript are somehow better at doing this the JavaScript is incorrect. Where they are better is in providing language mechanisms that allow you to precisely define the shape of the data. Despite the author's suggestions, they are not that much different than TypeScript. The only difference being that TypeScript chose to stay much closer to JavaScript for compatibility reasons. At the time that TypeScript came out, this was a reasonable requirement, since almost no-one wanted to move completely away from JavaScript. Five years later and the development world is ready for other languages. With WASM (Web Assembly) as a target (instead of just JavaScript), there are more possibilities than ever. JavaScript as a compile target is still open to runtime errors. When you use a higher-level language, you're restricting the range of functionality that you can use in the target bytecode/machine code. That is, when you write an if-statement in C, you're using the JMP statement, but you're only able to JMP to certain address locations instead of anywhere in addressable memory. It's the same with JavaScript as a compile target. It doesn't really matter that JavaScript allows too much -- what matters is what the higher-level language allows. TypeScript may still allow too much, but it's worlds better than JavaScript. It's true that PureScript or Elm or ClojureScript can close some loopholes that TypeScript leaves open. That's fine. But if you're going to just use JavaScript (or WASM) as a compile target, then why not choose a more-established language like C# or F#? ]]> https://www.earthli.com/news/view_article.php?id=3596 2019-01-08T22:24:05+01:00 https://earthli.com/users/marco

is about exactly what it sounds like it's about. Amid the flurry of comments with recommendations on how to pretend he (or she) knows how to build a web site by using tools... ]]>

is about exactly what it sounds like it's about. Amid the flurry of comments with recommendations on how to pretend he (or she) knows how to build a web site by using tools he's (or she's) never heard of, I chimed in with, What is it about software that makes people who have never done it think that they can do it professionally? What if your neighbor had heard you were a carpenter and had asked you to make a dining-room set for "good money"? Would you watch YouTube videos about how to make furniture and then charge money for the first furniture you ever made? What about if they'd asked for a haircut/trim/style/dye? Would you just go for it, after having asked around on /r/coiffeur for a few minutes? Or maybe they'd heard you were a chef and offered "good money" to cook their Thanksgiving dinner for them? Would you risk doing that? Probably not, because if you've never done any of those things, you're not good at them and charging for doing them can only backfire horribly. Unless your neighbor is a sap and a fool, in which case go for it. ]]> https://www.earthli.com/news/view_article.php?id=3642 2018-12-31T22:55:26+01:00 https://earthli.com/users/marco

is a wonderfully presented breakdown of how the path tracer found on a postcard does its magic. It's not super-fast (it takes 3 minutes to produce a much rougher version on the author's machine). He includes his final "cleaned-up source code" . [image] ... ]]>

is a wonderfully presented breakdown of how the path tracer found on a postcard does its magic. It's not super-fast (it takes 3 minutes to produce a much rougher version on the author's machine). He includes his final "cleaned-up source code" . [image] It comes from the same person who made the business card ray-tracer discussed in the article "Decyphering The Business Card Raytracer" by Fabien Sanglard . ]]> https://www.earthli.com/news/view_article.php?id=3636 2018-12-30T23:03:19+01:00 https://earthli.com/users/marco

is a wide-ranging rant about the inefficiency of C++ template programming and the degree to which it's inappropriate for many of the areas where C++ is used. Aras is one of the developers for the Unity game engine In particular, he highlights the disastrous... ]]>

is a wide-ranging rant about the inefficiency of C++ template programming and the degree to which it's inappropriate for many of the areas where C++ is used. Aras is one of the developers for the Unity game engine In particular, he highlights the disastrous compilation and execution speeds when using a lot of the STL. Not only that, but the debugging time is extremely slow, due to the inordinate amount of extra symbol information associated with hundreds of thousands of lines of code pulled in to implement relatively simple concepts that are standard in other languages, libraries and runtimes. On top of it all, even the high-level C++ code isn't very easy to read, despite the tremendous amount of abstraction. The optimized version of C++ code has an even worse compilation time, but it has a comparable/reasonable run-time to the C/C++-style version. However, it's very difficult to debug optimized code, which makes it doubly bad for development. Interactive development is hindered because of long compile times and, when debugging is necessary, most introspection tools don't work (e.g. reading variables) very well. It's the rare developer who can make headway debugging optimized code. He compares versions of an algorithm built using "classic" C/C++ programming vs. STL programming. He then compares to C#, which compiles and runs and debugs very quickly -- and is very easy to read, to boot. The problem with C++ boils down to its approach of making "everything a library". It's almost like an exercise in abstraction: since a few generic-programming concepts can be used to build everything in the library rather than the language, that's what C++ does. It's almost as if it does it to prove that it can be done. I'm all for removing redundancy in a language, but C++ is far from such a language. It's almost like the designers don't use their own language. He cites "Christer Ericson" "Goal of programmers is to ship, on time, on budget. It’s not “to produce code.” IMO most modern C++ proponents 1) overassign importance to source code over 2) compile times, debug[g]ability, cognitive load for new concepts and extra complexity, project needs, etc. 2 is what matters." Aras continues discussing the future of C++ and how it is currently used in game companies, for example. These are the companies using C++ the most. Rust is making some inroads, but the area is dominated by C/C++. Finally, he has some good advice for programmers -- for any professional, really -- on how to take criticism and turn it into something useful. "Ignoring literal trolls who complain on the internet “just for the lulz”, [the] majority of complaints do have [an] actual issue or problem behind it. It might be worded poorly, or exaggerated, or whoever is complaining did not think about other possible viewpoints, but there is a valid issue behind the complaint anyway. "What I do whenever someone complains about thing I’ve worked on, is try to forget about “me” and “work I did”, and get their point of view. What are they trying to solve, and what problems do they run into? The purpose of any software/library/language is to help their users solve the problems they have. It might be a perfect tool at solving their problem, an “ok I guess that will work” one, or a terribly bad one at that." As a postscript, the article "It is fast or it is wrong" by Nikita Tonsky discusses a very similar issue with Clojure vs. ClojureScript. "What do ClojureScript/Google Closure compilers do for so long? They are wasting your time, that’s what. Of course it’s nobody’s fault, but in the end, this whole solution is simply wrong. We can do the same thing much faster, we have proof of that, we have the means to do it, it just happens that we are not. But we could. If we wanted to. That huge overhead you’re paying, you’re paying it for nothing. You don’t get anything from being on JS, except a 2× performance hit and astronomical build times." I find these points interesting because programming is very much about which tools you use and how they help you to turn your work around more quickly. I'm in charge of choosing which languages, libraries and tools we use at Encodo and I'm hyper-aware of the efficiency losses when developers are hindered by their tools or libraries. Being the lead developer of our framework Quino makes me doubly aware of this. If you have a very slow feedback loop, then you'll take much longer to get your work done. I remember back in the late 90s/early 2000s, working with C++, where I would have to schedule builds because it took over 30 minutes to rebuild all of my static libraries if I made a low-level change. This was on a project that cross-compiled to Mac and Windows. Instead of working on my project, I spent way too much time massaging PCH files and avoiding making low-level changes so that I could continue testing. Bad tools that run too slowly are a problem. That's why you should always be very careful in choosing your languages, libraries and environments. Jumping ship to the "new hotness" very often means that you're going to have your time wasted by tools that aren't ready for prime time. ]]> https://www.earthli.com/news/view_article.php?id=3637 2018-12-30T22:12:44+01:00 https://earthli.com/users/marco

While you can use "Microsoft Docs" to find out which targets or platforms support which APIs, this resource lets you do it faster. You can browse a giant list of namespaces and click on any one of them to see the types, and then drill down to properties and methods. For each level, you can see a nice list of supported targets/platforms and the assemblies to use. You can also "Search", which opens what looks like a terminal that let's you camel-case search for your namespace, type or member. Selecting a result takes you to the location in the catalog. ".NET Framework Source-code browser" Yes, you read that correctly. I had no idea that this existed -- I've been digging through decompiled assembly code instead. This is much faster and includes the original documentation and comments. The source is syntax-highlighted and all types, methods and properties are linked. There's a document explorer, namespace explorer and project manager, all linked up very nicely. You can click any element and show all references in a separate pane. Clicking one of those references navigates there -- and other references in that file are also highlighted. If that's not sufficient, you can even download the entire source code as a ZIP file from here -- complete with solution and project files so you can open it in Visual Studio for browsing. ".NET Core Source Browser" This is the same thing as the link above, but for .NET Core sources. "FuGet" This is a NuGet package browser combined with an API browser over all of the assemblies in a package. * You can search the package for keywords. * You can browse types and see the formatted XML documentation. * You can diff versions of the package to see which APIs have changed. * You can jump to the code in the Git repository, if it's given in the package. It's an open-source GitHub project, so you could even run your own copy for diffing privately published packages. ]]> https://www.earthli.com/news/view_article.php?id=3557 2018-07-16T21:55:42+02:00 https://earthli.com/users/marco

() I used, application.RegisterSingle() This still didn't fix the crash because the mini-application doesn't get that registration automatically. I also can't use the same registration as above because that would just create a new unconfigured ApiSettings in each mini-application (the same as I had before, but now as a singleton). To go that route, I would have to replicate the configuration-loading for the ApiSettings as well. And I don't want to do that. Instead, I just injected the IApiSettings from the main application to the component responsible for creating the mini-application and registered the object as a singleton directly, as shown below. public class MiniApplicationFactory { public MiniApplicationFactory([NotNull] IApiSettings apiSettings) { if (apiSettings = null) { throw new ArgumentNullException(nameof(apiSettings(); } _apiSettings = apiSettings; } IApplication CreateApplication() { return new Application().UseRegisterSingle(_apiSettings); } [NotNull] private readonly IApiSettings _apiSettings; } On a side note, whereas C# syntax has become more concise and powerful from version to version, I still think it has a way to go in terms of terseness for such simple objects. For such things, Kotlin and TypeScript nicely illustrate what such a syntax could look like. [1] [Other Drawbacks] I mentioned above that this is only "one" of the reasons I don't like registering concrete singletons. The other two reasons are: 1. Complicates replacement: If the registered type is a concrete instance, then any replacement must inherit from this instance. The base class has to be constructed more carefully in order to allow for all foreseeable customizations. With an interface, the implementor is completely free to either use the existing class as a base or to re-implement the interface entirely. 2. Limits Mocking: Related to the first reason is that mocking is limited in its ability to override non-virtual methods. Even without a mocking library, you're just as hard-pressed to work around unwanted behavior in a hand-coded mock as you are with an actual replacement (as described above). Such limitations are non-existent with interfaces. -------------------------------------------------------------------------------- [1] I'm still waiting for C# to clean up a bit more of this syntax for me. The [NotNull] should be a language feature checked by the compiler so that the ArgumentNullException is no longer needed. On top of that, I'd like to see parameter properties, as in "TypeScript" (this is where you can prefix a constructor parameter with a keyword to declare and initialize it as a property). With a few more C#-language iterations that included non-nullable reference types and parameter properties, the example could look like the code below: public class MiniApplicationFactory { public MiniApplicationFactory(private IApiSettings apiSettings) { } IApplication CreateApplication() { return new Application().UseRegistereSingle(apiSettings); } } ]]> https://www.earthli.com/news/view_article.php?id=3546 2018-05-24T22:12:33+02:00 https://earthli.com/users/marco

shows you where we're headed. How do we plan to get there? A few years back, we made a big leap in Quino 2.0 to split up dependencies in anticipation of the initial release of .NET Core. Three tools were indispensable: "ReSharper" , "NDepend" and, of course, Visual Studio. Almost all... ]]>

shows you where we're headed. How do we plan to get there? A few years back, we made a big leap in Quino 2.0 to split up dependencies in anticipation of the initial release of .NET Core. Three tools were indispensable: "ReSharper" , "NDepend" and, of course, Visual Studio. Almost all .NET developers use Visual Studio, many use ReSharper and most should have at least heard of NDepend. At the time, I wrote a series of articles on the migration from two monolithic assemblies (Encodo and Quino) to dozens of layered and task-specific assemblies that allows applications to include our software in a much more fine-grained manner. As you can see from the articles, NDepend was the main tool I used for finding and tracking dependencies. [1] I used ReSharper to disentangle them. * "Splitting up assemblies in Quino using NDepend (Part I)" * "Iterating with NDepend to remove cyclic dependencies (Part II)" * "The Road to Quino 2.0: Maintaining architecture with NDepend (part I)" * "The Road to Quino 2.0: Maintaining architecture with NDepend (part II)" Since then, I've not taken advantage of NDepend's features for maintaining architecture as much as I'd like. I recently fired it up again to see where Quino stands now, with 5.0 in beta. But, first, let's think about why we're using yet another tool for examining our code. Since I started using NDepend, other tools have improved their support for helping a developer maintain code quality. * ReSharper itself has introduced tools for visualizing project and type dependencies with very nice graphs. However, there is currently no support for establishing boundaries and getting ReSharper to tell me when I've inadvertently introduced new dependencies. In fact, ReSharper's only improved its support for quickly pulling in a dependency with its excellent Nuget-Package integration. ReSharper is excellent for finding lower-level code smells, like formatting, style and null-reference issues, as well as language usage, missing documentation and code-complexity (with an extension). DotCover provides test-coverage data but I haven't used it for real-time analysis yet (I don't use continuous testing with ReSharper on Quino because I feel it would destroy my desktop). * Visual Studio has also been playing catch-up with ReSharper and has done an excellent job in the last couple of years. VS 2017 is much, much faster than its predecessors; without it, we would be foundering badly with a Quino solution with almost 150 projects. [2] Visual Studio provides Code Analysis and Portability Analysis and can calculate Code Metrics. Code Analysis is mostly covered by ReSharper, although it has a few extra inspections related to proper application and usage of the IDisposable pattern. The Portability Analysis is essential for moving libraries to .NET Standard but doesn't offer any insight into architectural violations like NDepend does. * We've recently started working with SonarQube on our TeamCity build server because a customer wanted to use it. It has a very nice UI and very nice reports, but doesn't go much farther than VS/R# inspections. Also, the report isn't in the UI, so it's not as quick to jump into the code. I don't want to review it here, since we only recently started working with it. It looks promising and is a welcome addition to that project. Hopefully more will reveal itself in time. * TeamCity provides a lot of the services that ReSharper also provides: inspections and code-coverage for builds. This takes quite a while, though, so we only run inspections and coverage for the Quino nightly build. The reports are nice but, as with SonarQube, of limited use because of the tenuous integration with Visual Studio. The integration works, but it's balky and we don't use it very much. Instead, we analyze inspections in real-time in Visual Studio with ReSharper and don't use real-time code-coverage. [3] * NDepend integrates right into Visual Studio and has a super-fast analysis with a very nice dashboard overview, from which you can drill down into myriad issues and reports and analyses, from technical debt (with very daunting but probably accurate estimates for repair) to type- and assembly-interdependency problems. NDepend can also integrate code-coverage results from DotCover to show how you're doing on that front on the dashboard as well. As with TeamCity and SonarQube, the analyses are retained as snapshots. With NDepend, you can quickly compare them (and comparing against a baseline is even included by default in the dashboard), which is essential to see if you're making progress or regressing. [4] NDepend also integrates with TeamCity, but we haven't set that up (yet). With a concrete .NET Core/Standard project in the wings/under development, we're finally ready to finish our push to make Quino Core ready for cross-platform development. For that, we're going to need NDepend's help, I think. Let's take a look at where we stand today. The first step is to choose what you want to cover. In the past, I've selected specific assemblies that corresponded to the "Core". I usually do the same when building code-coverage results, because the UI assemblies tend to skew the results heavily. As noted in a footnote below, we're starting an effort to separate Quino into high-level components (roughly, a core with satellites like Winform, WPF and Web). Once we've done that, the health of the core itself should be more apparent (I hope). For starters, though, I've thrown all assemblies in for both NDepend analysis as well as code coverage. Let's see how things stand overall. [image] The amount of information can be quite daunting but the latest incarnation of the dashboard is quite easy to read. All data is presented with a current number and a delta from the analysis against which you're comparing. Since I haven't run an analysis in a while, there's no previous data against which to compare, but that's OK. * Lines of Code * Code Elements (Types, Methods, etc.) * Comments (documentation) * Technical Debt * Code Coverage [5] * Quality Gates / Rules / Issues Let's start with the positive. * The Quino sources contain almost 50% documentation. That's not unexpected. The XML documentation from which we generate our developer documentation [6] is usually as long as or longer than the method itself. * We have a solid B rating for technical debt, which is really not bad, all things considered. I take that to mean that, even without looking, we instinctively produce code with a reasonable level of quality. Now to the cool part: you can click anything in the NDepend dashboard to see a full list of all of the data in the panel. Click the "B" on technical debt and you'll see an itemized and further-drillable list of the grades for all code elements. From there, you can see what led to the grade. By clicking the "Explore Debt" button, you get a drop-down list of pre-selected reports like "Types Hot Spots". Click lines of code and you get a breakdown of which projects/files/types/methods have the most lines of code Click failed quality gates to see where you've got the most major problems (Quino currently has 3 categories) [image] Click "Critical" or "Violated" rules to see architectural rules that you're violating. As with everything in NDepend, you can pick and choose which rules should apply. I use the default set of rules in Quino. [image] Most of our critical issues are for mutually-dependent namespaces. This is most likely not root namespaces crossing each other (though we'd like to get rid of those ASAP) but sub-namespaces that refer back to the root and vice-versa. This isn't necessarily a no-go, but it's definitely something to watch out for. There are so many interesting things in these reports: * Don't create threads explicitly (this is something we've been trying to reduce; I already knew about the one remaining, but it's great to see it in a report as a tracked metric) * Methods with too many parameters (you can adjust the threshold, of course) * Types too big: we'd have to check these because some of them are probably generated code, in which case we'd remove them from analysis. * Abstract constructors should be protected: ReSharper also indicates this one, but we have it as a suggestion, not a warning, so it doesn't get regularly cleaned up. It's not critical, but a code-style thing. I find the NDepend report much easier to browse than the inspection report in TeamCity. Click the "Low" issues (Quino has over 46,000!) and you can see that NDepend analyzes your code at an incredibly low level of granularity [image] * There are almost 10,000 cases where methods could have a lower visibility. This is good to know, but definitely low-priority. * Namespace does not correspond to file location: I'm surprised to see 4,400 violations because I thought that ReSharper managed that for us quite well. This one bears investigating – maybe NDepend found something ReSharper didn't or maybe I need to tweak NDepend's settings. Finally, there's absolutely everything, which includes boxing/unboxing issues [7], method-names too long, large interfaces, large instances (could also be generated classes). [image] These already marked as low, so don't worry that NDepend just rains information down on you. Stick to the critical/high violations and you'll have real issues to deal with (i.e. code that might actually lead to bugs rather than code that leads to maintenance issues or incurs technical debt, both of which are more long-term issues). What you'll also notice in the screenshots that NDepend doesn't just provide pre-baked reports: everything is based on its query language. That is, NDepend's analysis is lightning fast (takes only a few seconds for all of Quino) during which it builds up a huge database of information about your code that it then queries in real-time. NDepends provides a ton of pre-built queries linked from all over the UI, but you can adjust any of those queries in the pane at the top to tweak the results. The syntax is Linq to Sql and there are a ton of comments in the query to help you figure out what else you can do with it. [image] As noted above, the amount of information can be overwhelming, but just hang in there and figure out what NDepend is trying to tell you. You can pin or hide a lot of the floating windows if it's all just a bit too much at first. In our case, the test assemblies have more technical debt than the code that it tests. This isn't optimal, but it's better than the other way around. You might be tempted to exclude test assemblies from the analysis, to boost your grade, but I think that's a bad idea. Testing code is production code. Make it just as good as the code it tests to ensure overall quality. I did a quick comparison between Quino 4 and Quino 5 and we're moving in the right direction: the estimation of work required to get to grade A was already cut in half, so we've made good progress even without NDepend. I'm quite looking forward to using NDepend more regularly in the coming months. I've got my work cut out for me. -------------------------------------------------------------------------------- [1] Many thanks to Patrick Smacchia of NDepend for generously providing an evaluator's license to me over the years. [1] We came up with a plan for reducing the size of the core solution in a recent architecture meeting. More on that in a subsequent blog post. [1] Quino has 10,000 tests, many of which are integration tests, so a change to a highly shared component would trigger thousands of tests to run, possibly for minutes. I can't see how it would be efficient to run tests continuously as I type in Quino. I've used continuous testing in smaller projects and it's really wonderful (both with ReSharper and also Wallaby for TypeScript), but it doesn't work so well with Quino because of its size and highly generalized nature. [1] I ran the analysis on both Quino 4 and Quino 5, but wasn't able to directly compare results because I think I inadvertently threw them away with our nant clean command. I'd moved the ndepend out folder to the common folder and our command wiped out the previous results. I'll work on persisting those better in the future. [1] I generated coverage data using DotCover, but realized only later that I should have configured it to generate NDepend-compatible coverage data (as detailed in "NDepend Coverage Data" . I'll have to do that and run it again. For now, no coverage data in NDepend. This is what it looks like in DotCover, though. Not too shabby: [image] [1] Getting that documentation out to our developers is also a work-in-progress. Until recently, we've been stymied by the lack of a good tool and ugly templates. But recently we added "DocFX" support to Quino and the generated documentation is gorgeous. There'll be a post hopefully soon announcing the public availability of Quino documentation. [1] There's probably a lot of low-hanging fruit of inadvertent allocations here. On the other hand, if they're not code hot paths, then they're mostly harmless. It's more a matter of coding consistently. There's also an extension for ReSharper (the "Heap Allocations Viewer") that indicates allocations directly in the IDE, in real-time. I have it installed, and it's nice to see where I'm incurring allocations. ]]> https://www.earthli.com/news/view_article.php?id=3534 2018-03-31T23:28:27+02:00 https://earthli.com/users/marco

discusses several low-level and scholarly analyses of how common file-systems and user-space applications deal with read/write errors. * How theoretically consistent is the file system? * How well-documented are patterns that guarantee consistency? * How ... ]]>

discusses several low-level and scholarly analyses of how common file-systems and user-space applications deal with read/write errors. * How theoretically consistent is the file system? * How well-documented are patterns that guarantee consistency? * How well-understand are these patterns in the communities using them? * How do common applications (e.g. source control, databases, etc.) use these patterns? * Are these applications guaranteeing consistency? * What about the file-system designs? Are those airtight? * Are the file-system implementations correct? * How do the various components deal with hardware degradation or failure? [Asynchronous programming is hard] File-system operations work with devices and are thus asynchronous by nature. The analyses discovered similar ordering issues as with multi-threaded code. "The most common class of error was incorrectly assuming ordering between syscalls. The next most common class of error was assuming that syscalls were atomic2. These are fundamentally the same issues people run into when doing multithreaded programming. Correctly reasoning about re-ordering behavior and inserting barriers correctly is hard. But even though shared memory concurrency is considered a hard problem that requires great care, writing to files isn’t treated the same way, even though it’s actually harder in a number of ways." This is why most applications should use a framework or runtime support to access the file system. Even this might not be enough, though, if the implementation is still not robust enough for the application requirements. The .NET runtime has for quite a while now offered an API that uses async/await (i.e. a promise/future-based API), which at the very least indicates the asynchronous nature of these calls, with separate paths for success and error. This is better than nothing, even if the implementation occasionally fails to properly propagate errors (as we see with the POSIX APIs below). At any rate, the article drives home the point that programming against file systems is hard. "People almost always just run some tests to see if things work, rather than making sure they’re coding against what’s legal in a POSIX filesystem." Having a few tests is better than nothing, but it's even better to hoist your code up as many levels of abstraction as possible and avoid having to know about how to interleave fsync calls at all. Unless you're writing a database or a source-control system, right? [A common problem: documentation] He goes on to discuss "how much misinformation is out there" and that "it’s hard for outsiders to troll through a decade and a half of mailing list postings to figure out which ones are still valid and which ones have been obsoleted" This is a common problem that applies not just to low-level systems programming, but to any other programming problem. We have a surfeit of choice: just search online and you'll find something that matches what you searched. * Is the source authoritative? * Is the source even competent? * Is the source relevant? Or just kind of related? * Is the source current? Or outdated? * Are you in an echo chamber that feels authoritative but is just a bunch of low-skill developers at a local maximum when the real answer to your problem is elsewhere and is actually much more elegant? I recently ran into this phenomenon when learning Docker. Docker has changed and improved so much that the Internet is literally littered with old and overly complicated solutions to problems that either no longer exist or that can be solved with a simple one-liner in a configuration file. If you follow the instructions you find online, it's possible that you'll have something that works the way you want it to, but it's also very likely that you'll end up with a Frankenstein's Monster of a setup that kind of works but is fragile in unnecessary ways. [Drives are not infallible] From the article: "So far, we’ve assumed that the disk works properly, or at least that the filesystem is able to detect when the disk has an error via SMART or some other kind of monitoring. I’d always figured that was the case until I started looking into it, but that assumption turns out to be completely wrong." That sounds bad, of course. It's not something we user-space programmers ever really think about, is it? You read from a file, you write to a file, it works, right? And if it doesn't work (super-rare, right?), then the runtime throws an exception. If we assume that the runtime throws an exception, we're also assuming that the runtime is notified when an error occurs during a read or write operation. This was, apparently, not the case (at least in 2005-2008; we'll see improvements below). "In one presentation, one of the authors remarked that the ext3 code had lots of comments like “I really hope a write error doesn't happen here" in places where errors weren't handled. [...] NTFS is somewhere in between. The authors found that it has many consistency checks built in, and is pretty good about propagating errors to the user. However, like ext3, it ignores write failures." Ignoring write failures! That's kind of incredible, but if you've ever relied heavily on NTFS, you know that there are bugs in it. Sometimes files are just mysteriously locked and inaccessible until the system is rebooted. Why does the problem go away on reboot? NTFS is journaled and can recover its data, but it needs to be unmounted and checked. Instead of panicking, the write error is ignored. [1] "At this point, we know that it's quite hard to write files in a way that ensures their robustness even when the underlying filesystem is correct, the underlying filesystem will have bugs, and that attempting to repair corruption to the filesystem may damage it further or destroy it." [Replicating the results] The papers referenced in the first article are quite old (a decade or more) but the conclusions are still fascinating. Luu discusses the need for replicating the study and laments that "replications usually give little to no academic credit. This is one of the many cases where the incentives align very poorly with producing real world impact." Happily, Luu followed up with another post, called "File-system error-handling" that reproduces some of the original results with the 2017 versions of the file systems. This is an interesting study in its own right, discussing in detail interesting nuggets like the fact that "apfs doesn’t checksum data because “[apfs] engineers contend that Apple devices basically don’t return bogus data”." (from "APFS in Detail: Data Integrity" ). The second article concludes that "Filesystem error handling seems to have improved." Basic write errors are now propagated to user-space wherever possible (i.e. if the drive is not dead). However, "[m]ost filesystems don’t have checksums for data and leave error detection and correction up to userspace software." This is probably something that most user-space software developers never think about, but it's crucially important. Does your software assume that the file system will always throw an error? Or does it "just assume[...] that filesystems and disks don’t have errors"? [Abstract it away!] The first article concludes with a citation from "Butler Lampson" : "Lampson suggests that the best known general purpose solution is to package up all of your parallelism into as small a box as possible and then have a wizard write the code in the box." This is generally a good approach for anything complicated: programmers should use as high-level an API as possible for a given task. Problems like security, memory-allocation, file-system access, networking, asynchronous/parallel programming...these all fall into that category. Generally, the advice is, as usual, to get your requirements, make components that satisfy those requirements and include automated tests that verify that the components will continue to satisfy the requirements. As Lampson says, don't write code that's beyond you -- get a "wizard" to write it instead. That's what most of us do when we use the runtime provided with our language. [2] The best you can usually do is to abstract away access to external systems (including the file system) so that you can improve behavior later, should it be required. The budget and reliability constraints of a project don't always allow you to program perfectly safely. What you can do is to make sure that the system can be made safer later with a reasonable amount of effort. To be clear: don't be unnecessarily sloppy, but don't tank your project guaranteeing NASA-level safety where its not needed. So what does that mean? If you're programming on .NET, it means you should probably stay away from some constructs that you've previously considered safe and not worth wrapping, like File or Directory. Instead of using these directly, use them from an injected service. This level of abstraction is not difficult to enforce if introduced early in a project and will allow for improved testing anyway. If the filesystem is abstracted, components will no longer need their tests to actually write out files in order to work. As discussed above, this isn't to say that you jeopardize your deadline to abstract away every single file-system reference. For some applications, file-system access is so intrinsic as to be un-mockable (e.g. databases, source-control, etc.). However, your application is probably not one of those. It's likely that your application reads/writes files in a highly localizable manner that could be wrapped in a simple component. This advice is similar to the by-now common practice of not using the global DateTime.UtcNow. How can this be a problem? Well, if code uses an IClock component instead, then tests can adjust "now" to be a point in the past or future and test scheduling components more easily. It's an easy pattern to follow in new code that pays for itself the first time you need to reproduce a timing problem. [Avoiding Bugs] At the end of the second article, there's an interesting discussion of how to avoid these kind of bugs -- or just bugs, in general. "There's a very old debate over how to prevent things like this from accidentally happening." Better "tools or processes"? Be "better programmers"? Are tools like guardrails? Does it make sense to keep driving, bashing back and forth across the road, but happy that the guardrails are keeping us on the road at all? Would you do that in a car? Well, no. But, yes, if that's the best option? What's the other option? Just stop the car and don't go anywhere anymore? Or get out and walk? That analogy has been beaten to death -- and I don't think it's very appropriate (as you can see from my discussion about abstraction above). Tools and processes are better than nothing. Proper programming practices and patterns are, as well. If you train yourself to use tried-and-true patterns, then you automatically avoid common errors. * Use a language with static type-checking [3] * Abstract away interfaces to the system * Use non-nullable references wherever possible * Use immutable data wherever possible [4] * Segregate mutable data into dumb objects The point isn't to be able to say that "there are no bugs"; it's be able to say that "these tested bugs won't happen". The point is to use practices that avoid whole classes of problems. [What are better tools?] "Even better than a static analysis tool would be a language that makes it harder to accidentally forget about checking for an error." And now we come to the justification for some of the newer languages out there. Rust is such a language, which attempts to fix many of the shortcomings of C and C++ in the domain of allocating, sharing, modifying and freeing memory. For error-handling, the article "The Error Model" by Joe Duffy discusses a very interesting and promising approach taken by a Microsoft Research team with Midori, a 100%-managed version of Windows. The basic insight is to separate bugs from recoverable errors and unrecoverable errors. A bug is something the user-space application did wrong (e.g. passing a null reference to a method that expects only non-null references). A recoverable error is a validation error encountered when processing user input. An unrecoverable error is a file-read error in a base configuration file or a stack overflow or an out-of-memory error. For almost all software, file-system errors are something that should just be considered an unrecoverable error. There is no reason why most applications should attempt to continue when e.g. the main configuration cannot be loaded. Most applications don't even need to be able to recover from that. The problem occurs so rarely that you should just get a file out of backup. Lower-level applications like Git or PostgreSql have to take more care to deal with file-system errors [5], but your software most likely doesn't need to handle them. As discussed above, be aware that they can happen, abstract your code from the file-system so you can test error situations and improve handling where needed, but fail fast unless your project has a requirement to be able to recover in error conditions. Generally, no-one expects a user-space application to include robust file-recovery. It's expected, though, that the application detects when something is wrong and reports it, failing fast rather than just limping along and corrupting data. -------------------------------------------------------------------------------- [1] Anecdotally, it's definitely possible to get file corruption: I've had critical configuration files filled with NULL bytes after certain catastrophic operations. [1] Or why we use libraries for tough tasks (e.g. the immutable data-structures libraries for .NET, which provide a performant and correct implementation). [1] Not that I didn't say explicit types. I've used Swift and TypeScript for production code (and played with F#), all of which have inferred types. I still prefer specifying parameter and return types because it helps me localize errors immensely. For most methods, it helps me reason about the code more easily without giving away any flexibility. Still, that's a matter of taste (and, perhaps, familiarity); the point is that static type-checking is a good thing. [1] The article "You can't Rust that" by Armin Ronacher discusses how to use immutability in a language that provides pretty good support for it, Rust. He comes to the proper conclusion that you should "[c]onsider promoting new state instead of interior mutability", which is, not coincidentally, the same concept that the .NET immutable-collections library uses. [1] And if you look at the analysis in the first article, Git wasn't particularly radiant, but PostgreSql and Sqlite both did quite well. ]]> https://www.earthli.com/news/view_article.php?id=3419 2017-05-14T21:38:17+02:00 https://earthli.com/users/marco

is clean, easy-to-navigate and has very quick and accurate search results. [Initial Results] * Encodo.Core compiles (almost) without change. The only change required was to move project-description attributes that used to be in the AssemblyInfo.cs file to the project file instead (where they admittedly make much more sense). If you don't do this, the compiler complains about "[CS0579] Duplicate 'System.Reflection.AssemblyCompanyAttribute' attribute" and so on. * Encodo.Expressions references Windows.System.Media for Color and the Colors constants. I changed those references to System.Drawing and Color, respectively -- something I knew I would have to do. * Encodo.Connections references the .NET-Framework--only WindowsIdentity. I will have to move these references to a Encodo.Core.Windows project and move creation of the CurrentCredentials, AnonymousCredentials and UserCredentials to a factory in the IOC. * Quino.Meta references the .NET-Framework--only WeakEventManager. There are only two references and these are used to implement a CollectionChanged feature that is nearly unused. I will probably have to copy/implement the WeakEventManager for now until we can deprecate those events permanently. * Quino.Data depends on Quino.Meta.Standard, which references System.Windows.Media (again) as well as a few other things. The Quino.Meta.Standard potpourri will have to be split up. I discovered all of these things using just VS Code and the command-line build. It was pretty easy and straightforward. So far, porting to .NET Standard is a much more rewarding process than our previous attempt at porting to .NET Core. [The Game Plan] At this point, I had a shadow copy of a bunch of the core Quino projects with new project files as well as a handful of ad-hoc changes and commented code in the source files. While OK for investigation, this was not a viable strategy for moving forward on a port for Quino. I want to be able to work in a branch of Quino while I further investigate the viability of: * Targeting parts of Quino to .Net Standard 2.0 while keeping other parts targeting the lowest version of .NET Framework that is compatible with .NET Standard 2.0 (4.6.1). This will, eventually, be only the Winform and WPF projects, which will never be supported under .NET Standard. * Using the new project-file format for all projects, regardless of target (which IDEs can I still use? Certainly the latest versions of Visual Studio et. al.) To test things out, I copied the new Encodo.Core project file back to the main Quino workspace and opened the old solution in Visual Studio for Mac and JetBrains Rider. [IDE Pros and Cons] [Visual Studio for Mac] Visual Studio for Mac says it's a production release, but it stumbled right out of the gate: it failed to compile Encodo.Core even though dotnet build had compiled it without complaint from the get-go. Visual Studio for Mac claimed that OperatingSytem was not available. However, according to "the documentation" , Operating System is available for .NET Standard -- but not in .NET Core. My theory is that Visual Studio for Mac was somehow misinterpreting my project file. Update: After closing and re-opening the IDE, though, this problem went away and I was able to build Encodo.Core as well. Shaky, but at least it works now. [image]Unfortunately, working with this IDE remained difficult. It stumbled again on the second project that I changed to .NET Standard. Encodo.Core and Encodo.Expressions both have the same framework property in their project files -- netstandard2.0 -- but, as you can see in the screenshot to the left, both are identified as .NETStandard.Library but one has version 2.0.0-preview1-25301-01 and the other has version 1.6.1. I have no idea where there second version number is coming from -- it looks like this IDE is mashing up the .NET Framework version and the .NET Standard versions. Not quite ready for primetime. Also, the application icon is mysteriously the bog-standard MacOS-app icon instead of something more...Visual Studio-y. [JetBrains Rider EAP (April 27th)] JetBrains Rider built the assembly without complaint, just as dotnet build did on the command line. Rider doesn't didn't stumble as hard as Visual Studio for Mac, but it also didn't had problems building projects after the framework had changed. On top of that, it wasn't always so easy to figure out what to do to get the framework downloaded and installed. Rider still has a bit of a way to go before I would make it my main IDE. I also noticed that, while Rider's project/dependencies view accurately reflects .NET Standard projects, the "project properties" dialog shows the framework version as just "2.0". The list of version numbers makes this look like I'm targeting .NET Framework 2.0. Addtionally, Rider's error messages in the build console are almost always truncated. [image]The image to the right is of the IDE trying to inform me that Encodo.Logging (which was still targeting .NET Framework 4.5) cannot reference Encodo.Core (which references NET Standard 2.0). If you copy/paste the message into an editor, you can see that's what it says. [1] [Visual Studio Code] I don't really know how to get Visual Studio Code to do much more than syntax-highlight my code and expose a terminal from which I can manually call dotnet build. They "write about" Roslyn integration where "[o]n startup the best matching projects are loaded automatically but you can also choose your projects manually". While I saw that the solution was loaded and recognized, I never saw any error-highlighting in VS Code. The documentation does say that it's "optimized for cross-platform .NET Core development" and my projects targeted .NET Standard so maybe that was the problem. At any rate, I didn't put much time into VS Code yet. [Next Steps] 1. Convert all Quino projects to use the new project-file format and target .NET Framework. Once that's all running with the new project-file format, it will be much easier to start targeting .NET Standard with certain parts of the framework 2. Change the target for all projects to .NET Framework 4.6.1 to ensure compatibility with .NET Standard once I start converting projects. 3. Convert projects to .NET Standard wherever possible. As stated above, Encodo.Core already works and there are only minor adjustments needed to be able to compile Encodo.Expressions and Quino.Meta. 4. Continue with conversion until I can compile Quino.Schema, Quino.Data.PostgreSql, Encodo.Parsers.Antlr and Quino.Web. With this core, we'd be able to run the WebAPI server we're building for a big customer on a Mac or a Linux box. 5. Given this proof-of-concept, a next step would be to deploy as an OWIN server to Linux on Amazon and finally see a Quino-based application running on a much leaner OS/Web-server stack than the current Windows/IIS one. I'll keep you posted. [2] -------------------------------------------------------------------------------- [1] Encodo.Expressions.AssemblyInfo.cs(14, 12): [CS0579] Duplicate 'System.Reflection.AssemblyCompanyAttribute' attribute Microsoft.NET.Sdk.Common.targets(77, 5): [null] Project '/Users/marco/Projects/Encodo/quino/src/libraries/Encodo.Core/Encodo.Core.csproj' targets '.NETStandard,Version=v2.0'. It cannot be referenced by a project that targets '.NETFramework,Version=v4.5'. [1] Update: I investigated a bit farther and I'm having trouble using NETStandard2.0 from NETFramework462 (the Mono version on Mac). I was pretty sure that's how it's supposed to work, but NETFramework (any version) doesn't seem to want to play with NETStandard right now. Visual Studio for Mac tells me that Encodo.Core (NETStandard2.0) cannot be used from Encodo.Expressions (Net462), which doesn't seem right, but I'm not going to fight with it on this machine anymore. I'm going to try it on a fully updated Windows box next -- just to remove the Mono/Mac/NETCore/Visual Studio for Mac factors from the equation. Once I've got things running on Windows, I'll prepare a NETStandard project-only solution that I'll try on the Mac. ]]> https://www.earthli.com/news/view_article.php?id=3418 2017-05-01T21:42:56+02:00 https://earthli.com/users/marco

that I was rewriting the Encodo C# Handbook. The original was "published almost exactly nine years ago" . There were a few "more releases" as well as a few "unpublished" "chapters" . I finally finished a version that I think I can once again recommend to my employees... ]]>

that I was rewriting the Encodo C# Handbook. The original was "published almost exactly nine years ago" . There were a few "more releases" as well as a few "unpublished" "chapters" . I finally finished a version that I think I can once again recommend to my employees at "Encodo" . The major changes are: * The entire book is now a "Git Repository" . All content is now in Markdown. Pull requests are welcome. * I've rewritten pretty much everything. I removed a lot of redundancies, standardized formulations and used a much more economical writing style than in previous versions. * Recommendations now include all versions of C# up to 7 * There is a clearer distinction between general and C#-specific recommendations * There are now four main sections: "Naming" , "Formatting" , "Usage" and Best Practices, which is broken into "Design" , "Safe Programming" , "Error-handling" , "Documentation" and a handful of other, smaller topics. Here's the introduction: "The focus of this document is on providing a reference for writing C#. It includes naming, structural and formatting conventions as well as best practices for writing clean, safe and maintainable code. Many of the best practices and conventions apply equally well to other languages." Check out the "whole thing" ! Or download the "PDF" that I included in the repository. ]]> https://www.earthli.com/news/view_article.php?id=3391 2017-03-04T20:20:22+01:00 https://earthli.com/users/marco

void): IA; } class A implements IA { f = (action: (p: IB) => void): IA => { return this; } } Some notes on this example: * The shape of interface IB isn't relevant to the discussion. * The intent of interface IA is to require implementors to define a method named f that takes single parameter of type İ => void and returns IA. * The implementation A above satisfies this requirement. It doesn't do anything with parameter action but that's OK. * The definition of A.f() is what a naive user of TypeScript would assume was the only way of satisfying the requirement from IA [Oddly compatible lambdas] However, the following implementations of IA also compile. class A2 implements IA { f = (action: () => IB): IA => { return this; } } class A3 implements IA { f = (action: (p: IB) => IB): IA => { return this; } } class A4 implements IA { f = (action: () => void): IA => { return this; } } class A5 implements IA { f = (): IA => { return this; } } [Forcing incompatibility] The only one I tried that doesn't compile is shown below. class A6 implements IA { f = (action: (p: number) => void): IA => { return this; } } In this case, the TypeScript compiler rightly shows the following error: [image] Hovering over the class name A5 shows the following tooltip: Class 'A5' incorrectly implements interface 'IA'. Types of property 'f' are incompatible. Type '(action: (p: number) => void) => IA' is not assignable to type '(action: (p: IB) => void) => IA'. Types of parameters 'action' and 'action' are incompatible. Type '(p: IB) => void' is not assignable to type '(p: number) => void'. Types of parameters 'p' and 'p' are incompatible. Type 'number' is not assignable to type 'IB'. To summarize, the following types seem to be compatible with İ => void: * () => IB * İ => IB * () => void * No parameter at all [The nitty-gritty of TypeScript's type system] In a more strongly typed language like C#, it's clear that none of this would fly. But this is TypeScript, which defines its typing model on compatibility with the dynamic language JavaScript. It almost looks like the type of the lambda isn't part of the type signature of the method, which came as a quite a surprise to me (and also to my colleague, Urs, who is much more of a TypeScript expert than I am). But maybe we don't know enough about the TypeScript type system. Let's look at the "Type compatibility" documentation for TypeScript. This section starts off with a "Note on Soundness", which contains a note that suggests that what we have above is completely valid TypeScript. "The places where TypeScript allows unsound behavior were carefully considered, and throughout this document we’ll explain where these happen and the motivating scenarios behind them." The section Comparing two functions starts off explaining some rather surprising things about the type-compatibility of functions: for a function to be type-compatible with another function, the types of its parameters must match the types of the target type's parameters, but the number of parameters doesn't have to match. So if the target type has 4 parameters and the lambda to assign has 0 parameters, that lambda is compatible. From the manual: let x = (a: number) => 0; let y = (b: number, s: string) => 0; y = x; // OK x = y; // Error For return types, the matching behavior is opposite. That is, a "bigger" type that satisfies the expected return type is just fine. let x = () => ({name: "Alice"}); let y = () => ({name: "Alice", location: "Seattle"}); x = y; // OK y = x; // Error because x() lacks a location property [Reëxamining the oddly compatible lambdas] Armed with this new knowledge, let's see if the previously bizarre-seeming behavior is actually valid. To recap, the TypeScript compiler says that following signatures are compatible with İ => void: * f(() => IB): IA: this is compatible because the zero parameters conform by definition and any return type is OK because void is expected. * f(İ => IB): IA: this is compatible because the single parameter conforms and any return type is OK because void is expected. * f(() => void): IA: this is compatible because because the zero parameters conform by definition and any return type is OK because void is expected. * f() => IA: this one looks plain wrong at first, but the same logic applies to the whole function f(İ => void) => IA instead of to the lambda parameter for it. The interface expects a function f with a single parameter, returning IA. By the first rule above, a function with zero parameters satisfies that requirement. * f((number) => void): IA: This does not satisfy the requirement because number is not compatible with IB. * f(number): IA: This does not satisfy the requirement because number is not compatible with İ => void. * f(): void: This does not satisfy the requirement because while zero parameters is OK, the type void is smaller than IA. Well, it looks like there's nothing to see here, folks. The compiler is doing exactly what it's supposed to. Move along and get on with your day. Unfortunately, that means that TypeScript is going to be considerably less helpful for ensuring program correctness than I'd previously thought. In fact, the caveat about Typescript "allow[ing] unsound behavior [in] carefully considered [places]" seems a bit disingenuous because, to a programmer accustomed to something like C# or Java or Swift, this kind of type-enforcement for method compatibility cannot be relied upon to enforce much of anything. [Actual vs. Formal Arguments] When I read "OOSC2" a long time ago [1], I remember how Bertrand Meyer made the distinction between the formal type of an argument (the type in the method signature) and the actual type of an argument (the runtime type). The method-type--conformance rules for TypeScript make sense for actual arguments. They ensure compatibility with JavaScript. What's not clear to me is that this same logic be applied to formal arguments that are only available in TypeScript. If I declare a specific type signature in an interface, what are the odds that I want the wishy-washy JavaScript-friendly type rules for those situations? From an architect's point of view, it would certainly be nicer to have more strict type-checking for formal definitions. Since we don't have that, this very lenient type-compatibility renders type-checking for lambdas largely useless in interface declarations. The compiler won't be able to tell you that your implementation no longer matches the interface declaration because almost anything you write will actually match. -------------------------------------------------------------------------------- [1] I'm a nerd, I read all 1300 pages twice. ]]> https://www.earthli.com/news/view_article.php?id=3380 2017-03-04T00:06:33+01:00 https://earthli.com/users/marco

presented a short talk at "Voxxed Days 2017" this year, called "The truth about code reviews" . Sebastian and I also attended the rest of the conference. The following is a list of notes and reactions to the talks. ------------------------------------------------------------------------ " " ["Engineering You" ] " Martin Thompson -- "Video" " The keynote was about our place in... ]]>

presented a short talk at "Voxxed Days 2017" this year, called "The truth about code reviews" . Sebastian and I also attended the rest of the conference. The following is a list of notes and reactions to the talks. -------------------------------------------------------------------------------- " " ["Engineering You" ] " Martin Thompson -- "Video" " The keynote was about our place in the history of software engineering. Martin described us more as alchemists than engineers right now, a sentiment with which I can only agree. There is too little precision, too little reproducibility and too little focus on safety for use to qualify as engineers. He gave as an example the pride with which car companies brag about the hundreds of millions of lines of code they have running in software in their cars: a claim that should send shivers down your spine. We know how this software is written and how it is tested. "Quino" has fewer than 100,000 lines of code (about 85,000, at least 15% of which is obsolete) and we've been building that for almost 10 years. How a company whose main business is building automobiles guarantees safety and correctness of 300 million lines of code is beyond my comprehension. I would venture that they don't. Highly recommended talk. Very interesting. Lots of good history mixed with common-sense recommendations, like the following: * Code reviews * Iterative design * Be an engineer, not an alchemist * Automate * Test References: * "SOFTWARE ENGINEERING: Report on a conference sponsored by the NATO SCIENCE COMMITTEE Garmisch, Germany, 7th to 11th October 1968" (PDF) * "Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems" (PDF) -------------------------------------------------------------------------------- " " ["Swiss Transport in Real Time: Tribulations in the Big Data Stack" ] " "Alexandre Masselot" -- "Video" " He discussed a proof-of-concept transport-tracking application. Uses the SBB REST API for vehicle positions (using the same API as exposed for the app). Then there is the OpenData Transport API for station-board information, which provides details about delays. Everything is available as JSON with relatively straightforward data models. Uses Kafka to handle this real-time data pipeline (kind of like Chronicle, RabbitMQ or EasyMQ, but from Apache). The pipeline includes reformatting the data into the desired format (mostly eliding unwanted data), then store them in LogStash and then to ElasticSearch, which allows easy querying of the stored data. This type of data isn't fundamentally relational, so a document-based store is appropriate. The transformation also involves extrapolating the data that you're interested in from the data you obtained. For example, determining whether a train is stopped. E.g. are there x events with the same position? Is the position near a station? It was developed in Scala with Akka actors as well as the Play framework for REST. They represented all stations and trains with actors (objects). The actors are async and can run on any number of machines. After that comes Cassandra? Are they trying to use every possible technology? I'm losing track over here. Deployment on Docker. Also uses Zookeeper in another container for load-balancing/redundancy. OMG buzzwords. He asks: Why not a single application on a single server? Classic Java on Tomcat? It doesn't scale. It can only scale up, but not out. The actual solution feels like a lot of moving parts, but each part does a compartmentalized task, handing off to the next piece. It ends up being quite lightweight, using very little CPU overall. The simple, one-use components scale natively and relatively easily (LogStash, streaming, docker). The app server using Akka can be scaled, but it's here that you have to invest time to use the available fallback and clustering strategies. To render the data on the map, they used React to manage the data and d3.js to render. React is fast and scalable (but as Encodo has also discovered, that's not free either). Also, the client-side CPU usage is not insignificant, even with a lot of nodes. He also discussed UX and UI with tests. How to visualize possibly overlapping and differently sized elements at different zoom levels. Used Jupiter to analyze data and produce graphs. Conclusion: offload the parts of your application that aren't your core problem to external software and services. Things like managing data streams, transforming data, etc. Focus on your models and analyzing your data. * "Visualizing massive data streams: a public transport use case" * "D3.js transitions killed my CPU! A d3.js & pixi.js comparison" * "An invitation to reproducible computational research" -------------------------------------------------------------------------------- " " ["Functional data structures in Java" ] " Oleg Šelajev -- "Video" " He discussed how to build reusable structures that don't share mutable state (non-imperative vs. functional). * Classic standard libraries define mutable data structures, like lists, arrays, etc. These are not optimal for multi-tasking and asynchronous work. Mutable data structures produce side effects. * Void is a "code smell" because the only reason to call it is to cause a side effect. Prefer pure methods. * A functional data structure has to be immutable. * A functional data structure has to be persistent. This is similar to the first property, but it allows for a new structure to be created that is a mutation of the prior version. Obviously, we want to optimize storage here, reusing as much of the prior version as possible (instead of copying). * This is how mutation works, since we know that the prior version will never change, so it can be freely referenced. * Return values from methods on functional data structures are referentially transparent. You can cache the value without worrying that it will ever change or disappear. * This allows an application to work lock-free instead of guarding access to all possibly mutating methods. * It is easier to reason about functional (pure) data structures. Any discussion of data-structure design/implementation will naturally involve balancing performance vs. storage. The safety is baked-in, but performance is always a concern when working with immutable data structures, most especially when changing them. Even though the average call time for a method is nearly constant (as with most mutable structures), what if you call too many expensive operations and skew the average in real-world use? Well, you can combat this by leveraging the cachability of your collections (as defined above) as a way of memoizing (a well-known performance-optimization technique which carries with it possibly higher storage costs if you can't share the memoized instances very much.) In some cases, you can reason about performance in the following way: if you get to a situation where you would have to do an expensive operation (e.g. the reverse implicit in balancing head/tail of a queue), you can only get to this situation by having done n cheap operations first. So it is proven that the average is still constant time. Destructive behavior (like deque) looks different than mutable data structure. In those cases, the operation returns both the removed element as well as a reference to the queue that represents the new state of the queue. Tuple> Dequeue(); For maps, you need a concept called Zip that lets you quickly build a representation of the structure where the element viewed at a particular point in an existing structure is different. So even when a desired mutation would require alteration of a lot of the underlying structure, this operation allows reuse of a lot more of the structure than would otherwise be possible. The node can point to different parent and child nodes, referencing the new part of the structure while embedded in as much of the prior version as possible. "Object-oriented programming makes it easier to reason about moving parts. FUnctional programming makes it easier to minimize moving parts." References: * "What's new in purely functional data structures since Okasaki?" -------------------------------------------------------------------------------- " " ["Does diversity really matter?" ] " Sombra González and Brigitte Hulliger -- "Video" " This talk began by posing the following questions to the audience. * Do you work with women in a technical capacity? (My answer: No. The closest I've come was a programmer I trained as part of a group of 7 others for a customer.) * Can you remember having been in a meeting with two women or more? (My answer: A couple of project meetings over the last couple of years, but no-one in a technical capacity. Also some con conference calls, but neither of the two female participants were in a technical capacity.) Good questions. Good topic. Mostly well-presented, although the middle dragged a bit: Sombra envisioned a (near-)future where women are the same as men in a tech world, a meritocracy. It didn't add very much. As with everywhere else, the software industry has to figure out how to deal with long maternity leaves. Some countries have introduced "rainbow" leaves, which allow sharing of the time between partners, so if the partner is male, the industry has to deal with male absence as well. That will probably help increase acceptance of female leave, as it removes the distinction. For small companies, these kinds of extended leaves are a big hurdle because we can't so easily absorb so much missing capacity. We haven't improved at all in the last quarter-century: there have been proportionally fewer women in technical software positions every year since 1991. The quit rate is much higher (41%) than for men (17%). This is not primarily due to family concerns, though. It's mostly due to women not feeling comfortable in an industry where they're often the only female in a meeting, on a team or in a company. Reference: * "@WeShapeTech" -------------------------------------------------------------------------------- " " ["The truth about code reviews" ] " Sebastian Greulach -- "Video" " This talk is a reduced version of the code-review talk that Sebastian has been doing for "Encodo Systems" in both English and German over the last year. The presentation includes some statistics about the value of code reviews, a discussion of which benefits you can expect to get, which types of reviewers are likely to yield which benefits as well as Encodo's approach and advice for integrating code reviews into your development process. This was the most informative and amazing presentation at the entire show. All kidding aside, the room was packed and the ratings were quite good. There seemed to be a lot of interest in process. Reference: * "English slides (long form)" * "German slides (long form)" -------------------------------------------------------------------------------- " " ["How Modern SQL Databases Come up with Algorithms that You Would Have Never Dreamed Of" ] " Lukas Eder -- "Video" " This guy was supremely entertaining. He is the undisputed master of the animated and reaction GIF in presentations. Informative, spirited and very funny. * SQL is a 4GL. * It's a declarative language. * He shows off with a calculation of the Mandelbrot set with PostgreSql (but that's not the presentation) * He presents an example of how to address business needs (e.g. how much money per film per day). Shows how simple joins are in SQL * Then he shows how to do it with classic Java (which sucks). Basically, he shows how good SQL is by slagging on Java * He shows something that could be on "Annotatiomania" * At this point, the Java code is so long that "they can see our code from space." * Eager-loading is a code smell. You should actually be able to get the objects that you want in the form that you want. The optimal result type is the exact shape of the data that you want, not the ORM objects. [1] * "When does that ever happen? Changing requirements. Never." * He's talking about using SQL instead of code because you don't care about algorithms or storage types or caches -- let the database developers worry about that. They're good at it. And they love it. And the questions that you're answering are higher-level. * He discusses how Java streams look much, much nicer. But I think .NET Linq is even nicer ... and he doesn't mention that at all. So he shows how the more readable API is much better in Java ... but it's now exactly how SQL works. * The Java is now more readable, but it's still lazy-loading a ton of data you don't need. You're doing stuff on the client that the database would do much better. * Java Streams are so much uglier than Linq. They are forced to use explicitly typed Tuples (because there is still no var) and the tuple elements are unnamed (p1, p2, etc.) C# 6 is still like this, but C# 7 introduced named items for anonymous tuples. * The general-purpose languages force us to think about these things when they are not our programming domain. * Database also is capable of caching execution plans and optimizing subsequent queries. Prepared queries are da bomb. [2] * Any algorithm that produces the correct result is acceptable. It doesn't matter how you get there. * We don't really know what the database is really doing with a declarative syntax. We probably can't even guess at the optimizations that the clever database is doing. Use "Explain Query" to see the estimated plan and the actual plan (based on the actual data and current situation on the database instance). * The cardinality is a hint that indicates whether to use linear or logarithmic algorithm. This will also give a hint as the order that the database will load data (e.g. to reduce the dataset as quickly as possible before applies further joins, ordering and restrictions). * [image]Conclusion: let the database choose the algorithm based on the dataset available and the current state of the database. He shows an example with a histogram: how a query with one filter might use an index whereas a different filter might be more efficient just scanning the whole table (because 90% of the data is required anyway). The database can take disk-access speed into account. How can the developer predict which algorithm to use? The data and deployment environment isn't known at compile-time. So since you can't know and you're not the guy to decide, then you should offload that decision to the software that does know: the SQL database. What about latencies for remote data? Same thing. Let the database decide. * Unless you're the one writing the database. * The database is really good at this. It remembers how well its estimate matched the actual cost and it uses this to improve its execution plan. * Oracle can actually change the execution plan "in flight" if it sees during execution that an assumption was grievously wrong. * Also, SQL is functional: no side effects. * "Coders want to code; they want to do everything themselves." * Also: use production data whenever possible so you have a commensurate dataset size. * Joolambda is a product from his company. Also JOOQ. Looks like Linq, actually. But maybe it works better? Nice clean API which works with arbitrary result sets instead of fixed ORM objects. Can Quino learn something from it? I asked him after the talk where JOOQ gets its metadata and it generates it from the database schema. * Without hash joins there are a whole lot of algorithms that aren't available (MySQL). * Put business logic in the database, but be careful because how do you test it? I talked to Martin, Vlad and Lukas after the talk about testing and we agreed that databases should be immutable (Martin forbids the UPDATE statement in his projects, where he can) and then you basically have an immutable data structure in a separate process with a really powerful and efficient query languages over the graph. [3] * Locks are terribly complicated and performance is unpredictable. He says he's "lucky to only work with read-only databases. So much easier. So much fun." * Summary of chat after talk (some repeat from above): Chatted with Martin, Vlad and Lukas [4] after the talk about testing the database. Martin suggested that you don't use the update statement, only insert. Lukas responded similarly, saying that we should use SQL for read-only logical queries. Jooq has a metadata generator for analyzing your database so that you can query it. It doesn't define objects; you can only define the Tuple that you will return. That is pretty cool. Martin also pointed out that you could enforce immutability and store your data in an immutable, queryable graph by using the database. References: * "10 SQL Tricks That You Didn’t Think Were Possible" ("Video" ) * "Big-O Cheat-Sheet" -------------------------------------------------------------------------------- " " ["A practical introduction to Category Theory" ] " Daniela Sfregola -- "Video" " Category theory is about Monads, examples of which are Option, Try and Future (promise). The example she uses shows how to apply category-theory constructs to data-validation. The examples are in Scala, although the API that she presents looks very similar to the terminology used in Java's Streams API. E.g. flatMap(). That's Select() for C# developers. Similarly, Options is Nullable, although I can't think of the type analog for Some or None. Her validation example is well-made, going from returning an Option which is no better than a Boolean. Then she shows an Either but that doesn't allow for having both sides wrong. This can be done with Either but it's painful. That's why we invented pattern-matching (now available in C# 7). When she introduced a Validated, which is capable of returning a list of errors. "Focus on how things compose." The talk was quite short and didn't introduce much new. The pattern-matching syntax in Scala is a bit wordy. -------------------------------------------------------------------------------- " " ["g º f patterns" ] " Mario Fusco -- "Video" " Since my previous talk was done early, I joined Sebastian in this one. I saw only the tail-end of it, but man are the streams() libraries still really wordy. Welcome to functional programming, Java! Still, I'm disappointed that I can't use streams() in the Android project I'm working on because it required Java 8, which forces API level 24, which excludes a lot of devices. Sebastian said the talk was pretty good. -------------------------------------------------------------------------------- " " ["What about CSS? Progressive Enhancement and CSS" ] " Ire Aderinokun -- "Video" " * She's from Lagos, Nigeria. Google talk something or other. * Nicest slides I've seen all day. * Graceful degradation is the solution for only the current best browser. It doesn't necessarily scale to future versions. * Most designers test only one version older than the supported version. Encodo tests the versions required in the spec. * The goal isn't to dazzle the user, but to deliver the information to the widest possible audience. * Admittedly, some sites do have "dazzle the user" as a goal Rules: * Use sparse, semantic markup * Use plain text for the content in the markup * The basic layout should work without CSS * CSS is an enhancement * Enhanced behavior can be provided by unobtrusive JavaScript * End-user browser preferences should be respected (e.g. don't restrict zooming the UI, since a lot of users can't see so well) WTF is the squirrel browser? (It turns out it's "UC Browser" , popular in China.) Or the one with the strange globe? (Maybe Flock? Not sure.) Does Opera really have higher market-share than IE? Probably globally, right? Phone browser in India/China/etc. She showed a really cool graph of how many hours you have to work to use 500MB of data. Germany: 1h, Brazil: 56h, US: 6h. Bandwidth matters. A lot. WWW != Wealthy Western Web ammirite? * Use aria rules if you know that you might run on browsers that don't understand the new tag types (increasingly unlikely). Still, phone browsers in Africa probably have never heard of

or

. * There is no difference between an unsupported CSS property or a bad value or name in the style, selector, etc. * CSS doesn't have built-in fallbacks More rules: * Start with sensible HTML (same as above) * Go "Mobile-first" * Use media queries * Use flexbox (was designed as a progressive enhancement, so vertical-align is ignored when flexing is enabled.) * What about "Offline-first"? That is, making sure that your app works offline to at least some degree. Syncing data can be a pain, depending on the data, though. If you just have data to log, that's independent of other data, it's OK. * Use CSS Feature queries (detect support or NOT support) * Use progressive enhancement to deal with IE, which doesn't support feature queries * A good tip is that a property with a bad value is ignored by the browser. What about the future of the web? VR? Old devices handed down from the 1st to the 3rd world. I asked about testing that the progressive enhancements work as programmed, but no-one has any new ideas for testing, though. Manual testing to verify that the enhancements and fallbacks work. References: * "BitsOfCo.De" * "Understanding Progressive Enhancement" by Aaron Gustafson -------------------------------------------------------------------------------- " " ["I just hacked your app!" ] " Marcos Placona -- "Video" " He started off the talk as a bandit, reverse-engineering a Base64-encoded name/password. He used Charles to get MITM. It was a nice trick, and it probably works on a lot of devices and apps. It's very easy to make a hackable application if you don't think about security. He uses a nice word-definition slide with pronunciation and usage to make it look all official. * Pace is a bit slow at first. * Pokemon Hack was a MITM; it wasn't malicious: kids just didn't feel like walking. Important to remember that if motivation is high, a hacker will try really hard. * Beer hack was a loyalty hack (Kuba Gretsky) * Encrypt all the values instead of sending plain-text * But be careful of where you put your keys * This one guy Luke Chadwick uploaded his Amazon key to GitHub by accident. Farmers who watch every damned public commit got it, spun up some EC2 instances and started mining BitCoins * Use security features where possible * Use certificate pinning with the CertificatePinner() * Do NOT trust the device * Do NOT trust the app; it can be decompiled. * What about magic strings? * You can get your keys from a server * Or you can encrypt them, but what about the encryption key for the encryption key? * Get the key from the NDK. You can store information in the NDK itself, which is more secure and less decompilable than app code. * Check that the application name hasn't been changed. * Check that the package manager is supported/correct; otherwise, your app has been republished to a new server. * Or you can also check that the installer is Google or Amazon * Check your application signature; you can check whether the app was actually compiled by YOU * Check if the device is rooted (he used some exec() command) * Check for emulator (If the build fingerprint starts with "emulator") * Do not allow users to switch your App into debug mode * Use ProGuard, DexGuard. ProGuard is the lite version. DexGuard supports a lot of the checks listed above. DexGuard uses non-Latin Unicode for obfuscation. :-) Unfortunately, it's a per-user charge. That's per user of your app. * So, SafetyNet it is! That's more like it. * "The Internet is not a Safe Place" (shows a slide of a dirty van with "Hannah Montana Concert Shuttle" sprayed on the side) * Try to hack your own applications. Always. -------------------------------------------------------------------------------- [1] Just thinking out loud: Often, though, you want the object, more or less, so it's OK, no? Can we add this kind of anonymous-loading to Quino? It would be interesting to get arbitrary result sets. Like QQL. But it's nice to work with metadata (although we're very close to the SQL model anyway. Can we make it easier to build queries that aggregate?) [1] Quino doesn't use those either. Lots of wasted time there. Can we move toward working closer with the database API? How is it that we've avoided aggregations for so long? We just write those queries by hand. [1] Also thinking of a talk for next year: Schema Migration with Quino. I really like SQL and it's the best answer to a lot of questions. But I'm a programmer and I don't like to define my metadata in table form. I like to describe my model in an agnostic way so that I can re-use my metadata in more than just a database context. Or what about a focus on building metadata-based applications? Capturing your domain model in a way that's not so database-centric? [1] Nice clean API which works with arbitrary result sets instead of fixed ORM objects. ]]> https://www.earthli.com/news/view_article.php?id=3368 2017-02-06T00:10:55+01:00 https://earthli.com/users/marco

. If you read through the relatively long document, you can at least see that they're giving the... ]]>

. If you read through the relatively long document, you can at least see that they're giving the API design a tremendous amount of thought. [API Considerations for Strings] There are so many factors to weigh when building the API, especially for a low-level construct like String. * As they state right at the beginning of the document, they are concerned with "Ergonomics, Correctness, Performance" (probably in that order). * How does the API affect storage? * Is it still possible to use a COW pattern in order to save memory for multiple copies of the same string? Other, similar languages like C# and Java have slowly moved to a more-eager copying mechanism to reduce complexity in the memory-manager for strings, especially when used in multi-tasking. * How allocation-efficient is the base string library? Does the API help the more well-worn code paths avoid allocation unless absolutely necessary? * What about slicing support? Does the API force copying when it would not be needed? Does it at least allow the decision to copy to be delayed until absolutely necessary? * How accessible are the various supported representations? (E.g. UTF8 vs. UTF16) * How compatible/performant is the optimally ergonomic API with the "Objective-C interoperability" ? This is a common case and must be as close to allocation-free as possible and fast (because thunking between Swift code and Objective-C/Cocoa APIs is very common). * Does the API leverage patterns from other parts of the API rather than expanding the String API with a bunch of overloads? (E.g. the "discussion of storage for sub-strings" .) * Is immutable the default, with mutability opt-in? (This prevents unwanted copies and dangling references in the reference-counted world of Swift ... although Strings are actually structs rather than classes.) * Does the API do the "right thing" by default? In the case of Swift's string-handling, this means that the caller of the API works with Unicode graphemes, by default. * What about case-sensitive/insensitive comparisons? Accent sensitivity? * What about ordering? Collation? Localization? * Does the API scale nicely to allow increasing specificity, with good defaults? * Is there consistency within the string API? * What about consistency with similar constructs, like Array? * How does the API fit with with developer expectations? Should the String be a Collection? If so, what is the default item-type? * Why doesn't the Character have the same or a similar API as a String? (E.g. why can't you get the sub-structure of the grapheme cluster for a character without first casting it to a String?) [Slices/Substrings] A good example is the discussion of how to represent string slices: should there be a separate type, called Substring, analogous to the ArraySlice that already exists for an Array? "Long-term storage of Substring instances is discouraged. A substring holds a reference to the entire storage of a larger string, not just to the portion it presents, even after the original string's lifetime ends. "[...] "The downside of having two types is the inconvenience of sometimes having a Substring when you need a String, and vice-versa. It is likely this would be a significantly bigger problem than with Array and ArraySlice, as slicing of String is such a common operation. It is especially relevant to existing code that assumes String is the currency type -- that is, the default string type used for everyday exchange between APIs. To ease the pain of type mismatches, Substring should be a subtype of String in the same way that Int is a subtype of Optional." [To implement Collection or not?] For those that watch as the API for Swift evolves from one major version to another -- with each change introducing non--backward-compatible incompatibilities -- this document should hopefully reassure them that the changes are not made lightly. It may seem like the designers don't have a plan, but, over the years, designers and opinions change. E.g. Witness the "discussion of what the default representation of the string should be" . "[...] in Swift 1.0, String was a collection of Character (extended grapheme clusters). [...] In Swift 2.0, String's Collection conformance was dropped, because we convinced ourselves that its semantics differed from those of Collection too significantly." After listing several reasons why the change in Swift 2.0 was not a good direction, they conclude that in 4.0, they should revert to the original behavior. "It would be much better to legitimize the conformance to Collection and simply document the oddity of any concatenation corner-cases, than to deny users the benefits on the grounds that a few cases are confusing." Again, the discussion is open and public and, despite the claims of some who think that they're just a bunch of cowboys changing stuff willy-nilly, they have a documented plan. It's unfortunate that it took them so long to get there, but this kind of design isn't always easy. [Consolidating Index Types] Because Swift uses Unicode grapheme clusters as the default "items" view for strings, the discussion of "string indices" might seem unnecessarily abstract for developers coming from other languages, where the index is always an int int bytes. "String currently has four views--characters, unicodeScalars, utf8, and utf16 [...]" Because of these different views, it's necessary to discuss how to reduce API surface by consolidating the various index types used to refer to individual elements in these different "views" on a String. [Doing the Right Thing] It's not like C# -- and most other mainstream languages -- have anything to brag about with their string-handling. In that respect, even Swift 1 and 2 are light-years ahead in Unicode correctness with their focus on grapheme clusters rather than the utterly nonsensical 90s-era bytes still used in those other languages. The "Guidance for API Designers" shows how they try to build the API so that it makes sense for callers. "A Substring passed where String is expected will be implicitly copied. When compared to the "same type, copied storage" model, we have effectively deferred the cost of copying from the point where a substring is created until it must be converted to String for use with an API. "A user who needs to optimize away copies altogether should use this guideline: if for performance reasons you are tempted to add a Range argument to your method as well as a String to avoid unnecessary copies, you should instead use Substring." Their goal is noble, though it's unclear to what degree the vision can be realized. The "following citation" could be written as the high-level goal of any API. "We should represent these aspects as orthogonal, composable components, abstracting pattern matchers into a protocol like this one, that can allow us to define logical operations once, without introducing overloads, and massively reducing API surface area." -------------------------------------------------------------------------------- [1] Also as Microsoft did, Apple now has an "official, public issue tracker for Swift." [2] You could have knocked me over with a feather. [1] The downside: it's Atlassian Jira. ]]> https://www.earthli.com/news/view_article.php?id=3367 2017-02-04T18:17:03+01:00 https://earthli.com/users/marco

[2] and reported this issue as "A possible tuple-inference/parameter-resolution bug in Swift 3.0.1" [A Simple, Generic Function] We'll start off with a very basic example, shown below. [image] The example above shows a very simple function, generic in its single parameter with a required argument label a:. As expected, the compiler determines the generic type T to be Int. I'm not a big fan of argument labels for such simple functions, so I like to use the _ to free the caller from writing the label, as shown below. [image] As you can see, the result of calling the function is unchanged. [Or Maybe Not So Simple?] Let's try calling the function with some other combinations of parameters and see what happens. [image] If you're coming from another programming language, it might be quite surprising that the Swift compiler happily compiles every single one of these examples. Let's take them one at a time. * int: This works as expected * odd: This is the call that I experienced in my original code. At the time, I was utterly mystified how Swift -- a supposedly very strictly typed language -- allowed me to call a function with a single parameter with two parameters. This example's output makes it more obvious what's going on here: Swift interpreted the two parameters as a Tuple. Is that correct, though? Are the parentheses allowed to serve double-duty both as part of the function-call expression and as part of the tuple expression? * tuple: With two sets of parentheses, it's clear that the compiler interprets T as tuple (Int, Int). * labels: The issue with double-duty parentheses isn't limited to anonymous tuples. The compiler treats what looks like two labeled function-call parameters as a tuple with two Ints labeled a: and b:. * nestedTuple: The compiler seems to be playing fast and loose with parentheses inside of a function call. The compiler sees the same type for the parameter with one, two and three sets of parentheses. [3] I would have expected the type to be ((Int, Int)) instead. * complexTuple: As with tuple, the compiler interprets the type for this call correctly. [Narrowing Down the Issue] The issue with double-duty parentheses seems to be limited to function calls without argument labels. When I changed the function definition to require a label, the compiler choked on all of the calls, as expected. To fix the problem, I added the argument label for each call and you can see the results below. [image] * int: This works as expected * odd: With an argument label, instead of inferring the tuple type (Int, Int), the compiler correctly binds the label to the first parameter 1. The second parameter 2 is marked as an error. * tuple: With two sets of parentheses, it's clear that the compiler interprets T as tuple (Int, Int). * labels: This example behaves the same as odd, with the second parameter b: 2 flagged as an error. * nestedTuple: This example works the same as tuple, with the compiler ignoring the extra set of parentheses, as it did without an argument label. * complexTuple: As with tuple, the compiler interprets the type for this call correctly. [Swift Grammar] I claimed above that I was pretty sure that we're looking at a compiler bug here. I took a closer look at the productions for tuples and functions defined in The Swift Programming Language (Swift 3.0.1) manual available from Apple. First, let's look at tuples: [image] As expected, a tuple expression is created by surrounding zero or more comma-separated expressions (with optional identifiers) in parentheses. I don't see anything about folding parentheses in the grammar, so it's unclear why (((1))) produces the same type as (1). Using parentheses makes it a bit difficult to see what's going on with the types, so I'm going to translate to C# notation. * () => empty tuple [4] * (1) => Tuple * ((1)) => Tuple> * ...and so on. This seems to be a separate issue from the second, but opposite, problem: instead of ignoring parentheses, the compiler allows one set of parentheses to simultaneously denote the argument clause of a single-arity function call and an argument of type Tuple encompassing all parameters. A look at the grammar of a function call shows that the parentheses are required. [image] Nowhere did I find anything in the grammar that would allow the kind of folding I observed in the compiler, as shown in the examples above. I'm honestly not sure how that would be indicated in grammar notation. [Conclusion] Given how surprising the result is, I can't imagine this is anything but a bug. Even if it can be shown that the Swift compiler is correctly interpreting these cases, it's confusing that the type-inference is different with and without labels. -------------------------------------------------------------------------------- [1] The X-Code playground is a very decent REPL for this kind of example. Here's the code I used, if you want to play around on your own. func test(_ a: T) -> String { return String(describing: type(of: T.self)) } var int = test(1) var odd = test(1, 2) var tuple = test((1, 2)) var labels = test(a: 1, b: 2) var nestedTuple = test(((1, 2))) var complexTuple = test((1, (2, 3))) [1] I was amazed to find that Apple actually has a normal bug tracker for which I could create an account. Wonders never cease. [1] I didn't include the examples, but the type is unchanged with four, five and six sets of parentheses. The compiler treats them as semantically irrelevant, though the Swift grammar doesn't allow for this, as far as I could tell from the BNF in the official manual. [1] This is apparently legal in Swift, but I can't divine its purpose in an actual program ]]> https://www.earthli.com/news/view_article.php?id=3356 2017-01-15T23:40:49+01:00 https://earthli.com/users/marco

was an interesting analysis of a recent "stricter" trend in programming languages, as evidenced by Swift and Kotlin. I think TypeScript is also taking some steps along this path, as well as Rust, which I have a read a lot about, but haven't had much occasion to use. [Correct] ... ]]>

was an interesting analysis of a recent "stricter" trend in programming languages, as evidenced by Swift and Kotlin. I think TypeScript is also taking some steps along this path, as well as Rust, which I have a read a lot about, but haven't had much occasion to use. [Correct vs. Expressive] The point Martin makes is that all of these languages seem to be heedlessly improving correctness at the possible cost of expressiveness and maintainability. That is, as types are inferred from implementation, it can become more difficult to pinpoint where the intent of the programmer and the understanding of the compiler parted ways. As well, with increasing strictness -- e.g. non-null references, reference-ownership, explicit exceptions, explicit overrides -- there comes increasing overhead in maintaining code. Not only that, but developers must know their types -- and hence their design -- up front, which restricts evolving design as practiced in the very successful TDD approach and seems to be headed back to the stone age of waterfall design. As well, that level of strictness convinces developers -- who are similarly encouraged by the language designers -- that once their code compiles, then it runs as expected. But then they think they don't need to test, whereas the compiler really has no idea whether your code does what it should do. All it can guarantee is that no exception went unhandled -- or explicitly ignored -- (e.g. in Kotlin or Swift) or there are no race conditions or deadlocks (Rust) or that there are no null references where not explicitly programmed (Swift, Kotlin, TypeScript). These compiler-enforced language features are very useful, but are in the same class as the spell-checker in your text editor. Having no red, wavy lines in your document is no guarantee that the document makes any sense whatsoever. So these are interesting and useful features. They can lead to increased safety. But, they won't make your program do what it's supposed to do. At best, they help you avoid writing behavior that you most definitely don't want. These features are nice to have, but they are not worth having at any price. [Round Two: Defaming Types] It was an interesting article that I more-or-less agreed with. The follow-up article "Types and Tests" by Robert C. Martin followed close on its heels because Martin apparently wanted to respond to feedback he'd received on the first article. I thought he went a bit far in the second article. For example, he emphasized that, "No, types are not tests. Type systems are not tests. Type checking is not testing. Here’s why." That's absolutely true, but types are still related to testing. Types help me specify my interface more precisely and I can trust the compiler to enforce them. That's a lot of tests I don't have to write. Otherwise, for every API I write, I'd have to write tests to prove that only the supported types can be passed in -- and I'd also have to specify how my API behaves when value with an incorrect type is passed in. Do I fail silently? How do I let the caller know what to expect? This seems not only sloppy but time-consuming. It sounds like busy work, having to think about this kind of stuff for every API. [Precise Requirements] Martin continues, "[...] the way f is called has nothing to do with the required behavior of the system. Rather it is a test of an arbitrary constraint imposed by the programmer. A constraint that was likely over[-]specified from the point of view of the system requirements. (Emphasis added.)" The first sentence is a useful observation. The second is hyperbole. Indicating int rather than object for a parameter called limit hardly seems like an over-specification. In fact, in seems like exactly what I want. If the requirement says shall allow a user to enter a value for limit... rather than shall allow a user to enter a positive number for limit..., then I would argue that 99% of the time it's the requirement that isn't precise enough. I would not assume that the requirements engineer knew just what she was doing when she left the door open for a limit given as a string. Without types, our requirements would also become bloated with over-definitions like: * The system shall allow the user to enter a value that defines the upper limit of records to return. * The system shall throw an exception of type ArgumentOutOfRangeException for values that are less than zero or greater than 1000. * The system shall marshal the value to a numeric representation where necessary. * The system shall throw an exception of type ClassCastException if the given value cannot be marshaled to a numeric value. For this specification, a developer could write: public void SetLimit(object limit) { int limitAsNumber; if (!Int32.ParseInt(limit, out limitAsNumber)) { throw new ClassCastException("..."); } if (limitAsNumber > 1000) { throw new ArgumentOutOfRangeException("limit"); } _limit = limit; } The developer could also write: public void SetLimit(UInt32 limit) { if (limit > 1000) { throw new ArgumentOutOfRangeException("limit"); } _limit = limit; } That's actually what we want the developer to write, no? If you choose JavaScript to implement this requirement, then you would need to over-specify because you need to decide how to handle values with unsupported types. If the requirements engineer is allowed to assume that the implementing language has a minimal type system, then the requirements are also easier to write, as shown below. * The system shall allow the user to enter a positive integer that defines the upper limit of records to return. * The system shall throw an exception of type ArgumentOutOfRangeException for values that are less than zero or greater than 1000. * The system shall marshal the value to a numeric representation where necessary. * The system shall throw an exception of type ClassCastException if the given value cannot be marshaled to a numeric value. [Expressive Types are a Good Thing] Assuming a minimal type system in the target language saves time and effort. The requirements engineer can specify more concisely and the software engineer wastes less time writing boilerplate that has nothing to do with application behavior. Martin finished up with this sentiment, "So, no, type systems do not decrease the testing load. Not even the tiniest bit. But they can prevent some errors that unit tests might not see. (e.g. Double vs. Int) (Emphasis added.)" As you can imagine, I strongly disagree with the "[n]ot even the tiniest bit" part, based on my arguments above. If you use JavaScript, then you have to test all valid input and verify its behavior. In JavaScript, literally any data is valid input and it's up to your method to declare it invalid. Only tests can provide any protection against your method being called at runtime with invalid data. You have to write a test to verify that your method throws an error when passed a double rather than an int. Most people will not write these kind of tests, which I suspect is why Martin says there's no change in testing load. [Strict Languages for Bad Programmers?] I agree that the pendulum in Swift has swung too far in a restrictive direction. The language does feel pretty overloaded. I also agree that the behavior of the system itself needs to be tested and that types don't help you there. Martin again, "On the other hand, internal self-consistency does not mean the program exhibits the correct behavior. Behavior and self-consistency are orthogonal concepts. Well behaved programs can be, and have been, written in languages with high ambiguity and low internal consistency. Badly behaved programs have been written in languages that are deeply self-consistent and tolerate few ambiguities. (Emphasis added.)" Agreed. I think, though, that Martin might be forgetting about all of the people writing software who aren't the kind of people who can write a well-behaved program in a wildly inconsistent language. I, for example, am so awesome [1] that I wrote my entire web-site software in PHP -- one of the worst languages in the world for internal self-consistency -- and it's been running my site for going on 18 years. Programming skill and iron discipline fill the gap left by language consistency. But for bad programmers? They write utter garbage in PHP. Maybe it's not a bad idea to create languages that channel poorly disciplined programmers into better practices. I take the point from the previous article ("Dark Path" ) that bad programmers will simply work their way around the rigor, where possible. They will mark every class as open in Swift instead of thinking about their architecture. For those of us with discipline, the language will put up roadblocks that force us to write more code rather than less. [Unfixable Errors?] As a counterexample, there is Rust, which enforces reference-ownership in a way that guarantees concurrent code with no deadlocks and no race conditions. This is a good thing. It probably gets in your way when you're trying to write other types of programs, but it's overall a good thing. I haven't had any personal experience with it, but I've heard that it's sometimes difficult to figure out why a given program won't compile. I would hope that these situations become fewer with experience, but would also be cautious because I remember programming in C++ with templates and know how much time can be lost when you don't know how to fix your program based on an error message. [Non-nullable types are not a step too far] I, for one, like that my compiler tells me when I have potential null-reference exceptions. I use attributes in C# to tell me exactly that and I use R# to find all places in my code where I have potential violations. Those are more tests that I don't have to write, if the compiler can "prove" that this code is never called with a null reference. [2] It lets me write more concise implementation and spares me a lot of scaffolding. [What about const/non-mutable types?] Many years ago, I had the same experience with const in C++ as Martin discusses. After some time working with const, I starting making everything I possibly could const in order to eliminate a whole class of mutation errors in my code. That did have consequences, at the time. Changing one thing could -- as Martin describes for his hypothetical language TDP -- lead to knock-on changes throughout the code base. Generics can have this effect, as well, with changes leaking into all of the places they're used. I wrote a "blog series" on having pulled back from generics in a few central places in Quino. [Exceptions in method signatures] I often felt the way that Martin does about Java's throws declaration. I imagine that I'll start to feel the same about Swift's, as well. I read once about a "nice typing system in Midori" , the managed version of Windows created by Joe Duffy and team at Microsoft Research, that I felt I would like to try (no pun intended). [Toward more expressive types] Martin says that he uses both dynamically and statically typed languages. He acknowledges that certain extensions to the type system can be useful (but just that some languages have gone too far). I, too, think some innovations can be very helpful. I like immutables (types, declarations, whatever) because they let me reason better about my code. They let me eliminate unwanted code paths with the compiler rather than having to write more rote tests that I think even Martin will agree have nothing to do with the original specification or the behavior of my application. If I can mark something as readonly because I don't expect it to ever need to be changed, that's a little note I've left for future programmers that, should they want to modify that value, they will have to make sure to reason differently about the implementation. The value was never intended to be rewritten and there are no tests for that behavior. It's a nice way of reducing the scope of the implementation. It simultaneously restricts that scope, but that's a good thing. A program can, very quickly, do a lot of things that it should not do. I don't want to write tests for all of this stuff. I have neither the inclination nor the time -- nor the budget -- to write tests for things that I could instead eliminate entirely from the realm of possibility with a powerful type system. [Agreeing with Martin] I read up on Kotlin and saw a seminar on it last year. I, too, noticed that there seems to be an "everything but the kitchen sink" feel to it. It's the same feeling I get when I look at Scala's type system, though that one is less about restriction than about letting you do everything in 3 different ways. I've been reading through the Swift language guide and I'm getting the same feeling. It doesn't help that they have their own name and keyword for nearly every commonly known programming concept. You can use self. but the guide prefers just ., which takes some getting used to. finally? Nope. Use defer instead. [An example from TypeScript] To be honest, I'm also a bit dizzy at how quickly the TypeScript type system has gotten more and more complex. "TypeScript 2.1: keyof and Lookup Types" by Marius Schulz includes details on even more typing specifications that let you infer types from dynamic objects with flow-control analysis. I think this is quite an interesting approach, akin to more functional languages, like ML and F#, where return types are inferred and even parameter types are inferred. Swift has also gone a long way in this direction. Interfaces are replaced with non-inheritable types that describe the shape of data. Types can even be inferred by which fields you access within conditionals so that a single variable has a different inferred type depending on which path through the code it takes. It's all very exciting, but I wonder how much can be used correctly -- especially by the aforementioned crappy programmers. For example, this is the definition for the Object.entries() method from JavaScript. interface ObjectConstructor { // ... entries(o: T): [keyof T, T[K]][]; // ... } [Inferred method return-types] After having used languages that have explicit return types for methods, I'm still a bit at sea when I read TypeScript code without them. I find myself hovering identifiers to see which type was inferred for them by the real-time compilation. I agree that the code is cleaner, but maybe something's gone missing. It's harder to tell what the hell I'm supposed to pass in as a parameter or what the hell I get back from a function when the type can be a union of 3 or 4 other vaguely and sometimes ad-hoc--defined types. For example, a lot of code just constantly redefines the hash-table interface rather than just defining a type for it ... so the caller isn't restricted to implementing a specific interface. This is nice for library code, I guess, but it makes it harder to reason about the code because you don't have good names for types. This is an interesting enough experience for seasoned programmers; I can't even imagine how average or bad programmers deal with it. I see where Martin is coming from, that he's afraid of BDUF, something he's been fighting for years by arguing that you can design as you go if you'll just test your code as you write it. If you see that a parameter has to be an IHashMap, that's easier to understand than { [key: string]: any } or { [key: string]: T } where T is a completely different type. There are advantages and disadvantages. "Every step down that path increases the difficulty of using and maintaining the language. Every step down that path forces users of the language to get their type models “right” up front; because changing them later is too expensive. Every step down that path forces us back into the regime of Big Design Up Front." I agree with the sentiment, but I don't know if we're there yet. Martin argues that there is a balance and maybe I need more experience with the languages he's horrified about. He does write: "I think Java and C# have done a reasonable job at hovering near the balance point. (If you ignore the horrible syntax for generics, and the ridiculous proscription against multiple inheritance.)" ...which I agree with wholeheartedly. I have learned to live without multiple inheritance, but I regularly railed against its absence for decades. I have given up because the world has moved on. I would love to see proper contravariance and covariant return types and anchored types, but I've kind of given up on seeing that kind of stuff in a mainstream language, as well. Instead, I've drifted more toward immutable, stateless, functional style -- even in C#. I'm ogling F#. I'm working with Swift now and will do much more of that this year. -------------------------------------------------------------------------------- [1] Did you expect me to soften that brag down here, in this footnote? [1] I also use R#'s solution-wide analysis. ]]> https://www.earthli.com/news/view_article.php?id=3355 2017-01-11T08:47:45+01:00 https://earthli.com/users/marco

covers all of the points well, with screen shots but I just wanted to record my steps, collected into a tight list. Screenshots for most of these steps are available in the blog linked above. If you don't have a license for DataGrip, you can "download" ... ]]>

covers all of the points well, with screen shots but I just wanted to record my steps, collected into a tight list. Screenshots for most of these steps are available in the blog linked above. If you don't have a license for DataGrip, you can "download" a 30-day trial or you can "download" the JetBrains Rider EAP, which bundles it. Once Rider is released, you'll have to have a license for it, but -- for now -- you can use it for free. * Open SQL Server Configuration Manager to make sure that the TCP/IP protocol for SQL Server Express in the SQL Server Network Configuration is enabled. Restart the SQL Server Windows service if you made changes. * Make sure that the SQL Server Browser Windows service is running. * Open the Database tool window.Add a SQL Server (Microsoft) connection. Make sure the driver is downloaded. Fill out the settings as shown for QuinoTests below (click to expand). Note that the port has been cleared. [image] ]]> https://www.earthli.com/news/view_article.php?id=3313 2016-11-24T20:02:47+01:00 https://earthli.com/users/marco

use it sooner rather than later. [4] That said, here are the things I've noticed that are missing: * They claim that EditorConfig is included, but it didn't work for me. I found an EditorConfig extension that I installed, but I'd already set the configuration manually by then, so I'm not sure what effect it had. * I'd turned CodeLens off, but I still saw a rogue Git process. I haven't seen this process before or since, so I can only assume it came from 2017. * The StyleCop by JetBrains extension is not available yet, so that's a lot of hints and warnings missing. The maintainers of this extension write that "they will release as soon as ReSharper does" . * The standard Visual Studio TestRunner is still unusable. I can't imagine working without the ReSharper TestRunner. [5] Everything else seemed to work fine, which speaks well of both VS2017 and R#'s latest EAP. [Conclusion] * Xamarin Studio/Visual Studio for Mac: this won't be in the mix yet. It doesn't offer anything more than Rider on Mac. Plus, the Quino solution isn't ready for Mac development yet. Once .NET Standard 2.0 is ready and we can target Quino to that, we'll take another look. * Project Rider: I'll be keeping an eye on future releases of JetBrains's Project Rider, but won't use the current EAP version again (tests don't work reliably). * Visual Studio 2017: Once this is released, I feel good about installing it and releasing to other Encodo developers much sooner than with other versions. Especially since ReSharper's support is already so solid and once the StyleCop extension is available. Fingers crossed that the quality stays the same. -------------------------------------------------------------------------------- [1] I.e. go to Base/Inheriting implementations, and so on. [1] Although .NET Core 1.1 has been released, it seems that .NET Standard 2.0 will not be available until 2017. This isn't exactly unexpected, but we aren't going to move Quino onto a new platform target until this is released. It's possible that we could target .NET Core as well, but we burned our fingers once, so we're going to hold off until .NET Standard. [1] E.g. I've been having sporadic test failures when the runner is unable to load the latest FakeItEasy assembly in Visual Studio on my desktop -- but never my laptop or our TeamCity build server. A bunch of test failures in Rider were due to the same reason. [1] We usually wait until the first service pack -- at least. [1] Which has been -- thankfully -- stable since 2016.2. ]]> https://www.earthli.com/news/view_article.php?id=3271 2016-06-05T12:52:31+02:00 https://earthli.com/users/marco

. To cite from that article, "[...] the most important part of code is to think about how you’re writing it and what you’re building. You shouldn’t write a single line without thinking of the myriad ways in" ... ]]>

. To cite from that article, "[...] the most important part of code is to think about how you’re writing it and what you’re building. You shouldn’t write a single line without thinking of the myriad ways in which it must fit into existing code and the established patterns and practices." With that in mind, I saw another teaching opportunity this week and wrote up my experience designing an improvement to an existing API. [Requirements] Before we write any code, we should know what we're doing. [1] * We use aspects (IMetaAspects) in Quino to add domain-specific metadata (e.g. the IVisibleAspect controls element visibility) * Suppose we have such an aspect with properties A1...AN. When we set property A1 to a new value, we want to retain the values of properties A2...AN (i.e. we don't want to discard previously set values) * The current pattern is to call FindOrAddAspect(). This method does what it advertises: If an aspect with the requested type already exists, it is returned; otherwise, an instance of that type is created, added and returned. The caller gets an instance of the requested type (e.g. IVisibleAspect). * Any properties on the requested type that you want to change must have setters. * If the requested type is an interface, then we end up defining our interface as mutable. * Other than when building the metadata, every other use of these interfaces should not make changes. * We would like to be able to define the interface as read-only (no setters) and make the implementation mutable (has setters). Code that builds the metadata uses both the interface and the implementation type. Although we're dealing concretely with aspects in Quino metadata, the pattern and techniques outlined below apply equally well to other, similar domains. [The current API] A good example is the IClassCacheAspect. It exposes five properties, four of which are read-only. You can modify the property (OrderOfMagnitude) through the interface. This is already not good, as we are forced to work with the implementation type in order to change any property other than OrderOfMagnitude. The current way to address this issue would be to make all of the properties settable on the interface. Then we could use the FindOrAddAspect() method with the IClassCacheAspect. For example, var cacheAspect = Element.Classes.Person.FindOrAddAspect( () => new ClassCacheAspect() ); cacheAspect.OrderOfMagnitude = 7; cacheAspect.Capacity = 1000; For comparison, if the caller were simply creating the aspect instead of getting a possibly-already-existing version, then it would just use an object initializer. var cacheAspect = Element.Classes.Person.Aspects.Add( new ClassCacheAspect() { OrderOfMagnitude = 7; Capacity = 1000; } } This works nicely for creating the initial aspect. But it causes an error if an aspect of that type had already been added. Can we design a single method with all the advantages? [The new API] A good way to approach a new is to ask: How would we want the method to look if we were calling it? Element.Classes.Person.SetCacheAspectValues( a => { a.OrderOfMagnitude = 7; a.Capacity = 1000; } ); If we only want to change a single property, we can use a one-liner: Element.Classes.Person.SetCacheAspectValues(a => a.Capacity = 1000); Nice. That's even cleaner and has fewer explicit dependencies than creating the aspect ourselves. [Making it work for one aspect type] Now that we know what we want the API to look like, let's see if it's possible to provide it. We request an interface from the list of aspects but want to use an implementation to set properties. The caller has to indicate how to create the instance if it doesn't already exist, but what if it does exist? We can't just upcast it because there is no guarantee that the existing aspect is the same implementation. These are relatively lightweight objects and the requirement above is that the property values on the existing aspect are set on the returned aspect, not that the existing aspect is preserved. What if we just provided a mechanism for copying properties from an existing aspect onto the new version? var cacheAspect = new ClassCacheAspect(); var existingCacheAspect = Element.Classes.Person.Aspects.FirstOfTypeOrDefault(); if (existingCacheAspect != null) { result.OrderOfMagnitude = existingAspect.OrderOfMagnitude; result.Capacity = existingAspect.Capacity; // Set all other properties } // Set custom values cacheAspect.OrderOfMagnitude = 7; cacheAspect.Capacity = 1000; This code does exactly what we want and doesn't require any setters on the interface properties. Let's pack this away into the API we defined above. The extension method is: public static ClassCacheAspect SetCacheAspectValues( this IMetaClass metaClass, Action setValues) { var result = new ClassCacheAspect(); var existingCacheAspect = metaClass.Aspects.FirstOfTypeOrDefault(); if (existingCacheAspect != null) { result.OrderOfMagnitude = existingAspect.OrderOfMagnitude; result.Capacity = existingAspect.Capacity; // Set all other properties } setValues(result); return result; } So that takes care of the boilerplate for the IClassCacheAspect. It hard-codes the implementation to ClassCacheAspect, but let's see how big a restriction that is once we've generalized below. [Generalize the aspect type] We want to see if we can do anything about generalizing SetCacheAspectValues() to work for other aspects. Let's first extract the main body of logic and generalize the aspects. public static TConcrete SetAspectValues( this IMetaClass metaClass, Action copyValues, Action setValues ) where TConcrete : new, TService where TService : IMetaAspect { var result = new TConcrete(); var existingAspect = metaClass.Aspects.FirstOfTypeOrDefault(); if (existingAspect != null) { copyValues(result, existingAspect); } setValues(result); return result; } [Remove constructor restriction] This isn't bad, but we've required that the TConcrete parameter implement a default constructor. Instead, we could require an additional parameter for creating the new aspect. public static TConcrete SetAspectValues( this IMetaClass metaClass, Func createAspect, Action copyValues, Action setValues ) where TConcrete : TService where TService : IMetaAspect { var result = createAspect(); var existingAspect = metaClass.Aspects.FirstOfTypeOrDefault(); if (existingAspect != null) { copyValues(result, existingAspect); } setValues(result); return result; } [Just pass in the new aspect to use] Wait, wait, wait. We not only don't need to the new generic constraint, we also don't need the createAspect lambda parameter, do we? Can't we just pass in the object instead of passing in a lambda to create the object and then calling it immediately? public static TConcrete SetAspectValues( this IMetaClass metaClass, TConcrete aspect, Action copyValues, Action setValues ) where TConcrete : TService where TService : IMetaAspect { var existingAspect = metaClass.Aspects.FirstOfTypeOrDefault(); if (existingAspect != null) { copyValues(aspect, existingAspect); } setValues(aspect); return aspect; } That's a bit more logical and intuitive, I think. [Redefine original method] We can now redefine our original method in terms of this one: public static ClassCacheAspect SetAspectValues( this IMetaClass metaClass, Action setValues) { return metaClass.UpdateAspect( new ClassCacheAspect(), (aspect, existingAspect) => { result.OrderOfMagnitude = existingAspect.OrderOfMagnitude; result.Capacity = existingAspect.Capacity; // Set all other properties }, setValues ); } [Generalize copying values] Can we somehow generalize the copying behavior? We could make a wrapper that expects an interface on the TService that would allow us to call CopyFrom(existingAspect). public static TConcrete SetAspectValues( this IMetaClass metaClass, TConcrete aspect, Action setValues ) where TConcrete : TService, ICopyTarget where TService : IMetaAspect { return metaClass.UpdateAspect( aspect, (aspect, existingAspect) => aspect.CopyFrom(existingAspect), setValues ); } What does the ICopyTarget interface look like? public interface ICopyTarget { void CopyFrom(object other); } This is going to lead to type-casting code at the start of every implementation to make sure that the other object is the right type. We can avoid that by using a generic type parameter instead. public interface ICopyTarget { void CopyFrom(T other); } That's better. How would we use it? Here's the definition for ClassCacheAspect: public class ClassCacheAspect : IClassCacheAspect, ICopyTarget { public void CopyFrom(IClassCacheAspect otherAspect) { OrderOfMagnitude = otherAspect.OrderOfMagnitude; Capacity = otherAspect.Capacity; // Set all other properties } } Since the final version of ICopyTarget has a generic type parameter, we need to adjust the extension method. But that's not a problem because we already have the required generic type parameter in the outer method. public static TConcrete SetAspectValues( this IMetaClass metaClass, TConcrete aspect, Action setValues ) where TConcrete : TService, ICopyTarget where TService : IMetaAspect { return metaClass.UpdateAspect( aspect, (aspect, existingAspect) => aspect.CopyFrom(existingAspect), setValues ); } [Final implementation] Assuming that the implementation of ClassCacheAspect implements ICopyTarget as shown above, then we can rewrite the cache-specific extension method to use the new extension method for ICopyTargets. public static ClassCacheAspect SetCacheAspectValues( this IMetaClass metaClass, Action setValues) { return metaClass.UpdateAspect( new ClassCacheAspect(), setValues ); } This is an extension method, so any caller that wants to use its own IClassCacheAspect could just copy/paste this one line of code and use its own aspect. [Conclusion] This is actually pretty neat and clean: * We have a pattern where all properties on the interface are read-only * We have a pattern where an aspect can indicate how its values are to be copied from another instance. This is basically boilerplate, but must be written only once per aspect -- and it can be located right in the implementation itself rather than in an extension method. * A caller building metadata passes in a single lambda to set values. Existing values are handled automatically. * Adding support for more aspects is straightforward and involves very little boilerplate. -------------------------------------------------------------------------------- [1] You would think that would be axiomatic. You'd be surprised. ]]> https://www.earthli.com/news/view_article.php?id=3266 2016-05-21T10:58:43+02:00 https://earthli.com/users/marco

lately and one topic that we've started to tackle is design for architecture/APIs. Even if you're not officially a software architect -- designing and building entire systems from scratch -- every developer designs code, on some level. "[A]lways [B]e" ... ]]>

lately and one topic that we've started to tackle is design for architecture/APIs. Even if you're not officially a software architect -- designing and building entire systems from scratch -- every developer designs code, on some level. "[A]lways [B]e [D]esigning" There are broad guidelines about how to format and style code, about how many lines to put in a method, about how many parameters to use, and so on. We strive for Clean Code(tm). But the most important part of code is to think about how you're writing it and what you're building. You shouldn't write a single line without thinking of the myriad ways in which it must fit into existing code and the established patterns and practices. We've written about this before, in the two-part series called "Questions to consider when designing APIs" ("Part I" and "Part II" ). Those two articles comprise a long list of aspects of a design to consider. "First make a good design, then compromise to fit project constraints." Your project defines the constraints under which you can design. That is, we should still have our designer caps on, but the options available are much more strictly limited. But, frustrating as that might be, it doesn't mean you should stop thinking. A good designer figures out what would be optimal, then adjusts the solution to fit the constraints. Otherwise, you'll forget what you were compromising from -- and your design skills either erode or never get better. We've been calling this concept ABD -- Always Be Designing. [1] Let's take a closer, concrete look, using a recent issue in the schema migration for Quino. Hopefully, this example illustrates how even the tiniest detail is important. [2] [A bug in the schema migrator] We detected the problem when the schema migration generated an invalid SQL statement. ALTER TABLE "punchclock__timeentry" ALTER COLUMN "personid" SET DEFAULT ; As you can see, the default value is missing. It seems that there are situations where the code that generates this SQL is unable to correctly determine that a default value could not be calculated. The code that calculates the default value is below. result = Builder.GetExpressionPayload( null, CommandFormatHints.DefaultValue, new ExpressionContext(prop), prop.DefaultValueGenerator ); To translate, there is a Builder that produces a payload. We're using that builder to get the payload (SQL, in this case) that corresponds to the DefaultValueGenerator expression for a given property, prop. This method is an extension method of the IDataCommandBuilder, reproduced below in full, with additional line-breaks for formatting: public static string GetExpressionPayload( this IDataCommandBuilder builder, [CanBeNull] TCommand command, CommandFormatHints hints, IExpressionContext context, params IExpression[] expressions) { if (builder == null) { throw new ArgumentNullException("builder"); } if (context == null) { throw new ArgumentNullException("context"); } if (expressions == null) { throw new ArgumentNullException("expressions"); } return builder.GetExpressionPayload( command, hints, context, expressions.Select( e => new ExecutableQueryItem(new ExecutableExpression(e)) ) ); } This method does no more than to package each item in the expressions parameter in an ExecutableQueryItem and call the interface method. The problem isn't immediately obvious. It stems from the fact that each ExecutableQueryItem can be marked as Handled. The extension method ignores this feature, and always returns a result. The caller is unaware that the result may correspond to an only partially handled expression. [Is there a quick fix?] Our first instinct is, naturally, to try to figure out how we can fix the problem. [3] In the code above, we could keep a reference to the executable items and then check if any of them were unhandled, like so: var executableItems = expressions.Select( e => new ExecutableQueryItem(new ExecutableExpression(e)) ); var result = builder.GetExpressionPayload(command, hints, context, executableItems); if (executableItems.Unhandled().Any()) { // Now what? } return result; } We can detect if at least one of the input expressions could not be mapped to SQL. But we don't know what to do with that information. * Do we throw an exception? No, we can't just do that. None of the callers are expecting an exception, so that's an API change. [4] * Do we return null? What can we return to indicate that the input expressions could not be mapped? Here we have the same problem as with throwing an exception: all callers assume that the result can be mapped. So there's no quick fix. We have to change an API. We have to design. [Part of the result is missing] As with most bugs, the challenge lies not in knowing how to fix the bug, but in how to fix the underlying design problem that led to the bug. The problem is actually not in the extension method, but in the method signature of the interface method. Instead of a single result, there are actually two results for this method call: * Can the given expressions be mapped to a string (the target representation)? * If so, what is that text? Instead of a Get method, this is a classic TryGet method. [How to Introduce the Change] If this code is already in production, then you have to figure out how to introduce the bug fix without breaking existing code. If you already have consumers of your API, you can't just change the signature and cause a compile error when they upgrade. You have to decorate the existing method with [Obsolete] and make a new interface method. So we don't change the existing method and instead add the method TryGetExpressionPayload() to IDataCommandBuilder. [What are the parameters?] Now, let's figure out what the parameters are going to be. The method called by the extension method above has a slightly different signature. [5] string GetExpressionPayload( [CanBeNull] TCommand command, CommandFormatHints hints, [NotNull] IExpressionContext context, [NotNull] IEnumerable> expressions ); That last parameter is a bit of a bear. What does it even mean? The signature of the extension method deals with simple IExpression objects -- I know what those are. But what are ExecutableQueryItems and IExecutableExpressions? As an author and maintainer of the data driver, I know that these objects are part of the internal representation of a query as it is processed. But as a caller of this method, I'm almost never going to have a list of these objects, am I? Let's find out. Me: Hey, ReSharper, how many callers of that method are there in the entire Quino source? ReSharper: Just one, Dave. [6] So, we defined an API with a signature that's so hairy no-one calls it except through an extension method that makes the signature more palatable. And it introduces a bug. Lovely. We've now figured out that our new method should accept a sequence of IExpression objects instead of ExecutableQueryItem objects. How's the signature looking so far? bool TryGetExpressionPayload( [CanBeNull] TCommand command, CommandFormatHints hints, [NotNull] IExpressionContext context, [NotNull] IEnumerable expressions, out string payload ); [Are We Done?] Not quite. There are two things that are still wrong with this signature, both important. [Fix the Result Type] One problem is that the rest of the IDataCommandBuilder deals with a generic payload type and this method only works for builders where the target representation is a string. The Mongo driver, for example, uses MongoStorePayload and MongoRetrievePayload objects instead of strings and throws a NotSupportedException for this API. That's not very elegant, but the Mongo driver was forced into that corner by the signature. Can we do better? The API would currently require Mongo to always return false because our Mongo driver doesn't know how to map anything to a string. But it could map to one of the aforementioned object representations. If we change the out parameter type from a string to an object, then any driver, regardless of payload representation, has at least the possibility of implementing this API correctly. [Fix parameters] Another problem is that the order of parameters does not conform to the code style for Encodo. * We prefer to place all non-nullable parameters first. Otherwise, a call that passes null as the first parameter looks strange. The command can be null, so it should move after the two non-nullable parameters. If we move it all the way to the end, we can even make it optional. * Also, primitives should come after the references. (So hints should be third.) * Also, semantically, the call is getting the payload for the expressions not the context. The first parameter should be the target of the method; the rest of the parameters provide context for that input. * The original method accepted params IExpression[]. Using params allows a caller to provide zero or more expressions, but it's only allowed on the terminal parameter. Instead, we'll accept an IEnumerable, which is more standard for the Quino library anyway. The final method signature is below. bool TryGetExpressionPayload( [NotNull] IEnumerable expressions, [NotNull] IExpressionContext context, CommandFormatHints hints, out object payload, [CanBeNull] TCommand command = default(TCommand) ); [Our API in Action] The schema migration called the original API like this: result = Builder.GetExpressionPayload( null, CommandFormatHints.DefaultValue, new ExpressionContext(prop), prop.DefaultValueGenerator ); return true; The call with the new API -- and with the bug fixed -- is shown below. The only non-functional addition is that we have to call ToSequence() on the first parameter (highlighted). Happily, though, we've fixed the bug and only include a default value in the field definition if one can actually be calculated. object payload; if (Builder.TryGetExpressionPayload( prop.DefaultValueGenerator.ToSequence(), new ExpressionContext(prop), CommandFormatHints.DefaultValue, out payload) ) { result = payload as string ?? payload.ToString(); return true; } [One More Design Decision...] A good rule of thumb is that if you find yourself explaining something in detail, it might still be too complicated. In that light, the call to ToSequence() is a little distracting. [7] It would be nice to be able to map a single expression without having to pack it into a sequence. So we have one more design decision to make: where do we add that method call? Directly to the interface, right? But the method for a single expression can easily be expressed in terms of the method we already have (as we saw above). It would be a shame if every implementor of the interface was forced to produce this boilerplate. Since we're using C#, we can instead extend the interface with a static method, as shown below (again, with more line breaks for this article): public static bool TryGetExpressionPayload( [NotNull] this IDataCommandBuilder builder, // Extend the builder [NotNull] IExpression expression, [NotNull] IExpressionContext context, CommandFormatHints hints, out object payload, [CanBeNull] TCommand command = default(TCommand) ) { return builder.TryGetExpressionPayload( expression.ToSequence(), context, hints, out payload, command ); } We not only avoided cluttering the interface with another method, but now a caller with a single expression doesn't have to create a sequence for it [8], as shown in the final version of the call below. object payload; if (Builder.TryGetExpressionPayload( prop.DefaultValueGenerator, new ExpressionContext(prop), CommandFormatHints.DefaultValue, out payload) ) { result = payload as string ?? payload.ToString(); return true; } [Conclusion] We saw in this post how we always have our designer/architect cap on, even when only fixing bugs. We took a look at a quick-fix and then backed out and realized that we were designing a new solution. Then we covered, in nigh-excruciating detail, our thought process as we came up with a new solution. Many thanks to Dani for the original design and Sebastian for the review! -------------------------------------------------------------------------------- [1] This is a bit of a riff on ABC -- Always Be Closing -- as popularized by Alec Baldwin in the movie "Glengarry Glen Ross" . [1] Also, understand that it took much longer to write this blog post and itemize each individual step of how we thought about the issue. In reality, we took only a couple of minutes to work through this chain of reasoning and come up with the solution we wanted. It was only after we'd finished designing that I realized that this was a good example of ABD. [1] Actually, our first instinct is to make sure that there is a failing test for this bug. But, this article deals with how to analyze problems and design fixes, not how to make sure that the code you write is tested. That's super-important, too, though, just so you know. Essential, even. [1] Even though C# doesn't include the exceptions thrown in the signature of a method, as Java does. Where the Java version is fraught with issues, see the "Recoverable Errors: Type-Directed Exceptions" chapter of "Midori: The Error Model" by Joe Duffy for a really nice proposal/implementation of a language feature that includes expected exceptions in the signature of a method. [1] Which is why we defined the extension method in the first place. [1] I'm fully aware that my name isn't Dave. It's just what ReSharper calls me. "Old-school reference." [1] This was pointed out, by the way, by a reviewer of this blog post and escaped the notice of both designers and the code-reviewer. API design is neither easy nor is it done on the first try. It's only finished after multiple developers have tried it out. Then, you'll probably be able to live with it. [1] Most developers would have used new [] { expression }, which I think is kind of ugly. ]]> https://www.earthli.com/news/view_article.php?id=3262 2016-05-12T22:22:36+02:00 https://earthli.com/users/marco

is also available for those with access to the Encodo issue tracker. [Highlights] * Lots of bug fixes and improvements for the Winform UI and German translations with the release of Punchclock ... ]]>

is also available for those with access to the Encodo issue tracker. [Highlights] * Lots of bug fixes and improvements for the Winform UI and German translations with the release of Punchclock on this version. ("QNO-5162" , "QNO-5159" , "QNO-5158" , "QNO-5157" , "QNO-5156" , "QNO-5140" , "QNO-5155" , "QNO-5145" , "QNO-5111" , "QNO-5107" , "QNO-5106" , "QNO-5104" , "QNO-5015" ) * DateTimeExtensions.GetDayOfWeek() had a leap-day bug ("QNO-5051" ) * Fixed how the hash code for GenericObjects is calculated, which fixes sorting issues in grids, specifically for non-persisted or transient objects ("QNO-5137" ) * Improvements to the IAccessControl API for getting groups and users and testing membership ("QNO-5133" ) * Add support for query aliases (e.g. for joining the same table multiple times) ("QNO-531" ) This changes the API surface only minimally. Applications can pass an alias when calling the Join method, as shown below, query.Join(Metadata.Project.Deputy, alias: "deputy") You can find more examples of aliased queries in the TestAliasedQuery(), TestJoinAliasedTables(), TestJoinChildTwice() defined in the QueryTests testing fixture. * Add a standalone IQueryAnalyzer for optimizations and in-memory mini-drivers ("QNO-4830" ) [Breaking changes] * ISchemaManager has been removed. Instead, you should retrieve the interface you were looking for from the IOC. The possible interfaces you might need are IImportHandler, IMappingBuilder, IPlanBuilder or ISchemaCommandFactory. * ISchemaManagerSettings.GetAuthorized() has been moved to ISchemaManagerAuthorizer. * The hash-code fix for GenericObjects may have an effect on the way your application sorts objects.The IParticipantManager (base interface of IAccessControl) no longer has a single method called GetGroups(IParticipant). This method was previously used to get the groups to which a user belongs and the child groups of a given group. This confusing double duty for the API led to an incorrect implementation for both usages. Instead, there are now two methods: * IEnumerable GetGroups(IUser user): Gets the groups for the given user * IEnumerable GetChildGroups(IGroup group): Gets the child groups for the given group The old method has been removed from the interface because (A) it never worked correctly anyway and (B) it conflicts with the new API. ]]> https://www.earthli.com/news/view_article.php?id=3261 2016-05-12T22:16:43+02:00 https://earthli.com/users/marco

: Proof of concept with metadata, PostgreSql (data and schema-migration) and Winforms UI * "1.0" : First customer product with PostgreSql, DevExpress Winforms UI and Reporting * "1.0.5" : MS-SQL driver (parity with PostgreSql driver) * "1.5.0" : Remoting data driver; require .NET 4.0 * "1.6.0" : Mongo/NoSQL data driver * "1.8.0" : Rewrite data driver to use sessions * "1.8.5" : Support "improved metadata-generation pattern" * "1.9.0" : Add plugin/overlay support * "1.10.0" : Require .NET 4.5; add JSON-based remoting protocol; Windows-service support * "1.13.0" : Rewrite security API * "v2.0-beta1" : Rewrite configuration, logging and schema-migration APIs * "v2.0-beta2" : Add V2 generated-code format * "2.0" : Finish configuration/IOC rewrite; produce NuGet packages for delivery * "2.2" : Stabilize Winform; support aliased tables in queries * "3.0" : Rewrite MetaBuilder API; improve support for plugins We took 1.5 years to get to v1. The initial major version was to signify the first time that Quino-based code went into external production. [2] After that, it took 6.5 years to get to v2. Although we added several large products that use Quino, we were always able to extend rather than significantly change anything in the core. The second major version was to signify sweeping changes made to address technical debt, to modernize certain components and to prepare for changes coming to the .NET platform. It took just 5 months to get to v3 for two reasons: 1. Although we were able to make a lot of planned changes in v2 [3], we had to leave some breaking changes for future versions. [4] 2. We now strictly adhere to the rule that a breaking change anywhere in the software's API -- and Quino's API surface is large -- leads automatically to a major-version change. [5] [Roadmap] So that's where we've been. Where are we headed? As you can see above, Quino is a very mature product that satisfies the needs of a wide array of software on all tiers. What more is there to add? Quino's design has always been driven by a combination of customer requirements and what we anticipated would be customer requirements. We're currently working on the following features. Modeling improvements This work builds on the API changes made to the MetaBuilder in v3. We're creating a more fluent, modern and extensible API for building metadata. We hope to be able to add these changes incrementally without introducing any breaking changes. [6] WPF / VSG A natural use of the rich metadata in Quino is to generate user interfaces for business entities without have to hand-tool each form. From the POC onward, Quino has included support for generating UIs for .NET Winforms. Winforms has been replaced on the Windows desktop with WPF and UWP. We've gotten quite far with being able to generate WPF applications from Quino metadata. The screenshots below come from a pre-alpha version of the Sandbox application included in the Quino solution. [image][image][image][image] You may have noticed the lovely style of the new UI. [7] We're using a VSG designed for us by "Ergosign" , for whom we've done some implementation work in the past. .NET Core If you've been following Microsoft's announcements, things are moving quickly in the .NET world. There are whole new platforms available, if you target your software to run on them. We're investigating the next target platforms for Quino. Currently that means getting the core of Quino -- Quino.Meta and its dependencies -- to compile under .NET Core. [image]As you can see in the screenshot, we've got one of the toughest assemblies to compile -- Encodo.Core. After that, we'll try for running some tests under Linux or OS X. The long-term goal is to be able to run Quino-based application and web servers on non-Windows -- and, most importantly, non-IIS -- platforms. [8] These changes will almost certainly cause builds using previous versions to break. Look for any additional platform support in an upcoming major-version release. -------------------------------------------------------------------------------- [1] There were, of course, more minor and patch releases throughout, but those didn't introduce any major new functionality. [1] Punchclock, our time-entry and invoicing software -- and Quino ""dogfood" (When a developer uses their own code for their own daily needs. Being a user as well as a developer creates the user empathy that is the hallmark of good software.)" product -- had been in use internally at Encodo earlier than that. [1] E.g. splitting the monolithic Encodo and Quino assemblies into dozens of new, smaller and much more focused assemblies. Reorganizing configuration around the IOC and rewriting application startup for more than just desktop applications was another sweeping change. [1] One of those breaking changes was to the MetaBuilder, which started off as a helper class for assembling application metadata, but became a monolithic and unavoidable dependency, even in v2. In v3, we made the breaking changes to remove this component from its central role and will continue to replace its functionality with components that more targeted, flexible and customizable. [1] In the years between v1 and v2, we used the minor-version number to indicate when breaking changes could be made. We also didn't try as hard to avoid breaking changes by gracefully deprecating code. The new approach tries very hard to avoid breaking changes but accepts the consequences when it's deemed necessary by the team. [1] That is, when users upgrade to a version with the newer APIs, they will get obsolete warnings but their existing code will continue to build and run, as before the upgrade. In this way, customers can smoothly upgrade without breaking any builds. [1] You may also have noticed that the "Sandbox Dialog View" includes a little tag in it for the "XAML Spy", a tool that we use for WPF development. Just so you know the screenshots aren't faked... :-) [1] As with the WPF interface, we're likely to dogfood all of these technologies with Punchclock, our time-tracking and invoicing system written with Quino. The application server and web components that run on Windows could be migrated to run on one of our many Linux machines instead. ]]> https://www.earthli.com/news/view_article.php?id=3260 2016-05-12T22:11:29+02:00 https://earthli.com/users/marco

is also available for those with access to the Encodo issue tracker. [Highlights] * Metadata builders are more lightweight now and have fewer restrictions on their API. ("QNO-4994" ) * Improve ... ]]>

is also available for those with access to the Encodo issue tracker. [Highlights] * Metadata builders are more lightweight now and have fewer restrictions on their API. ("QNO-4994" ) * Improve support for plugins ("QNO-5189" , "QNO-5190" , "QNO-5196" ) * Improve and clean up expression library ("QNO-5174" , "QNO-5176" , "QNO-5182" , "QNO-5183" , "QNO-5184" , "QNO-5188" , "QNO-5198" , "QNO-5205" ) * Improve schema-migration tools ("QNO-5179" , "QNO-5206" , "QNO-5212" , "QNO-5213" ) [Breaking changes] * IDataSession and IApplication now directly implement the IServiceRequestHandler and helper methods that used to extend IApplication now extend this interface instead, so calls like GetModel() can now be executed against an IApplication or an IDataSession. Many methods have been moved out of the IServiceRequestHandler interface to extension methods declared in the Encodo.IOC namespace. This move will require applications to update the usings. ReSharper will automatically find the correct namespace and apply it for you. * Similarly, the extension method ApplicationExtensions.GetInstance() has been replaced with a direct implementation of the IServiceRequestHandler by IApplication. * MetaBuilder.Include() has been replaced with Dependencies.Include() * When you call the new version of CreateModel(), you can no longer call CreateMainModule() because the main module is set up automatically. Although the call is marked as obsolete, it can only be combined with the older overload of the CreateModel(). Using it with the newer overload will cause a runtime error as the main module is added to the model twice. * The various methods to create paths with the MetaBuilder have been replaced by AddPath(). To rewrite a path, use the following style: Builder.AddPath( Elements.Classes.A.FromOne("Id"), Elements.Classes.B.ToMany("FileId"), path => path.SetMetaId(new Guid("...")).SetDeleteRule(MetaPathRule.Cascade), idx => idx.SetMetaId(new Guid("...")) ); ]]> https://www.earthli.com/news/view_article.php?id=3256 2016-04-26T21:40:40+02:00 https://earthli.com/users/marco

in 2008. At the time, we also published to several other standard places and got some good, positive feedback. Over the next year, I made some more changes and published new versions. The latest version is 1.5.2 and is... ]]>

in 2008. At the time, we also published to several other standard places and got some good, positive feedback. Over the next year, I made some more changes and published new versions. The latest version is 1.5.2 and is available from "Encodo's web site" . Since then, though I've made a few extra notes and corrected a few errors, but never published an official version again. This is not because Encodo hasn't improved or modernized its coding guidelines, but because of several issues, listed below. * At 72 pages, it's really quite long * A more compact, look-up reference would be nice * It contains a mix of C#-specific, Encodo-specific and library-specific advice * It's maintained in Microsoft Word * Code samples are manually formatted * New versions are simply new copies in versioned folders (no source control) * Collaboration is nearly impossible * There is nothing about any .NET version newer than 3.5 * There is no mention of any other programming language (e.g. TypeScript, JavaScript) * A lot of stuff is overly complicated (e.g. var advice) or just plain wrong (e.g. var advice) To address these issues and to accommodate the new requirements, here's what we're going to do: * Convert the entire document from Word to Markdown and put it in a Git repository * Collaboration? Pull requests. Branches. * Versioning? Standard diffing of commits. * Code samples? Automatic highlighting from GitLab (Encodo's internal server) or GitHub (external repository). * Separate the chapters into individual files and keep them shorter and more focused on a single topic * * Separate all of the advice and rules into the following piles: * General programming advice and best practices * C#-specific * Encodo-specific * Library-specific (e.g. Quino) These are the requirements and goals for a new version of the C# handbook. The immediate next steps are: 1. Convert current version from Microsoft Word to Markdown (done) 2. Add everything to a Git repository (done) 3. Overhaul the manual to remove incorrect and outdated material; address issues above (in progress) 4. Mirror externally ("GitHub" or "GitLab" or both) I hope to have an initial, modern version ready within the next month or so. ]]> https://www.earthli.com/news/view_article.php?id=3240 2016-04-07T22:27:10+02:00 https://earthli.com/users/marco

() .Register(); return result; } That doesn't look too bad, though, does it? It doesn't seem like it would cramp anyone's style too much, right? Aren't we being a bit nitpicky here? That's exactly why Quino 2.0 was released with this API. However, here we are, months later, and I've written a lot more configuration code and it's really starting to chafe that I have to declare a local variable and sort my method invocations. So I think it's worth addressing. Anything that disturbs me as the writer of the framework -- that gets in my way or makes me write more code than I'd like -- is going to disturb the users of the framework as well. Whether they're aware of it or not. [Developers are the Users of a Framework] In the best of worlds, users will complain about your crappy API and make you change it. In the world we're in, though, they will cheerfully and unquestioningly copy/paste the hell out of whatever examples of usage they find and cement your crappy API into their products forever. Do not underestimate how quickly calls to your inconvenient API will proliferate. In my experience, programmers really tend to just add a workaround for whatever annoys them instead of asking you to fix the problem at its root. This is a shame. I'd rather they just complained vociferously that the API is crap rather than using it and making me support it side-by-side with a better version for usually feels like an eternity. Maybe it's because I very often have control over framework code that I will just not deal with bad patterns or repetitive code. Also I've become very accustomed to having a wall of tests at my beck and call when I bound off on another initially risky but in-the-end rewarding refactoring. If you're not used to this level of control, then you just deal with awkward APIs or you build a workaround as a band-aid for the symptom rather than going after the root cause. [Better Sooner than Later] So while the code above doesn't trigger warning bells for most, once I'd written it a dozen times, my fingers were already itching to add [Obsolete] on something. I am well-aware that this is not a simple or cost-free endeavor. However, I happen to know that there aren't that many users of this API yet, so the damage can be controlled. If I wait, then replacing this API with something better later will take a bunch of versions, obsolete warnings, documentation and re-training until the old API is finally eradicated. It's much better to use your own APIs -- if you can -- before releasing them into the wild. Another more subtle reason why the API above poses a problem is that it's more difficult to discover, to learn. The difference in return types will feel arbitrary to product developers. Code-completion is less helpful than it could be. It would be much nicer if we could offer an API that helped users discover it at their own pace instead of making them step back and learn new concepts. Ideally, developers of Quino-based applications shouldn't have to know the subtle difference between the IOC and the application. [A Better Way] Something like the example below would be nice. return new Application() .UseStandard() .RegisterSingle() .UseOtherComponent() .Register(); Right? Not a gigantic change, but if you can imagine how a user would write that code, it's probably a lot easier and more fluid than writing the first example. In the second example, they would just keep asking code-completion for the next configuration method and it would just be there. [Attempt #1: Use a Self-referencing Generic Parameter] In order to do this, I'd already created an issue in our tracker to parameterize the IServiceRegistrationHandler type in order to be able to pass back the proper return type from registration methods. I'll show below what I mean, but I took a crack at it recently because I'd just watched the very interesting video "Fun with Generics" by Benjamin Hodgson , which starts off with a technique identical to the one I'd planned to use -- and that I'd already used successfully for the IQueryCondition interface. [2] Let's redefine the IServiceRegistrationHandler interface as shown below, public interface IServiceRegistrationHandler { TSelf Register() where TService : class where TImplementation : class, TService; // ... } Can you see how we pass the type we'd like to return as a generic type parameter? Then the descendants would be defined as, public interface IApplication : IServiceRegistrationHandler { } In the video, Hodgson notes that the technique has a name in formal notation, "F-bounded quantification" but that a snappier name comes from the C++ world, "curiously recurring template pattern". I've often called it a self-referencing generic parameter, which seems to be a popular search term as well. This is only the first step, though. The remaining work is to update all usages of the formerly non-parameterized interface IServiceRegistrationHandler. This means that a lot of extension methods like the one below public static IServiceRegistrationHandler RegisterCoreServices( [NotNull] this IServiceRegistrationHandler handler) { } will now look like this: public static TSelf RegisterCoreServices( [NotNull] this IServiceRegistrationHandler handler) where TSelf : IServiceRegistrationHandler { } This makes defining such methods more complex (again). [3] in my attempt at implementing this, Visual Studio indicated 170 errors remaining after I'd already updated a couple of extension methods. [Attempt #2: Simple Extension Methods] Instead of continuing down this path, we might just want to follow the pattern we established in a few other places, by defining both a Register method, which uses the IServiceRegistrationHandler, and a Use method, which uses the IApplication Here's an example of the corresponding "Use" method: public static IApplication UseCoreServices( [NotNull] this IApplication application) { if (application == null) { throw new ArgumentNullException("application"); } application .RegisterCoreServices() .RegisterSingle(application.GetServices()) .RegisterSingle(application); return application; } Though the technique involves a bit more boilerplate, it's easy to write and understand (and reason about) these methods. As mentioned in the initial sentence of this article, the cognitive load is lower than the technique with generic parameters. The only place where it would be nice to have an IApplication return type is from the Register* methods defined on the IServiceRegistrationHandler itself. We already decided that self-referential generic constraints would be too messy. Instead, we could define some extension methods that return the correct type. We can't name the method the same as the one that already exists on the interface [4], though, so let's prepend the word Use, as shown below: IApplication UseRegister( [NotNull] this IApplication application) where TService : class where TImplementation : class, TService; { if (application == null) { throw new ArgumentNullException("application"); } application.Register(); return application; } That's actually pretty consistent with the other configuration methods. Let's take it for a spin and see how it feels. Now that we have an alternative way of registering types fluently without "downgrading" the result type from IApplication to IServiceRegistrationHandler, we can rewrite the example from above as: return new Application() .UseStandard() .UseRegisterSingle() .UseOtherComponent() .UseRegister(); Instead of increasing cognitive load by trying to push the C# type system to places it's not ready to go (yet), we use tiny methods to tweak the API and make it easier for users of our framework to write code correctly. [5] -------------------------------------------------------------------------------- [1] See Encodo’s configuration library for Quino "Part 1" , "Part 2" and "Part 3" as well as API Design: Running and Application "Part 1" and "Part 2" and, finally, "Starting up an application, in detail" . [1] The video goes into quite a bit of depth on using generics to extend the type system in the direction of dependent types. Spoiler alert: he doesn't make it because the C# type system can't be abused in this way, but the journey is informative. [1] As detailed in the links in the first footnote, I'd just gotten rid of this kind of generic constraint in the configuration calls because it was so ugly and offered little benefit. [1] If you define an extension method for a descendant type that has the same name as a method of an ancestor interface, the method-resolution algorithm for C# will never use it. Why? Because the directly defined method matches the name and all the types and is a "stronger" match than an extension method. Perhaps an example is in order: interface IA { IA RegisterSingle(); } interface IB : IA { } static class BExtensions { static IB RegisterSingle(this IB b) { return b; } static IB UseStuff(this IB b) { return b; } } Let's try to call the method from BExtensions: public void Configure(IB b) { b.RegisterSingle().UseStuff(); } The call to UseStuff cannot be resolved because the return type of the matched RegisterSingle method is the IA of the interface method not the IB of the extension method. There is a solution, but you're not going to like it (I know I don't). public void Configure(IB b) { BExtensions.RegisterSingle(b).UseStuff(); } You have to specify the extension-method class's name explicitly, which engenders awkward fluent chaining -- you'll have to nest these calls if you have more than one -- but the desired method-resolution was obtained. But at what cost? "The horror...the horror." [1] The final example does not run against Quino 2.2, but will work in an upcoming version of Quino, probably 2.3 or 2.4. ]]> https://www.earthli.com/news/view_article.php?id=3235 2016-03-25T13:41:54+01:00 https://earthli.com/users/marco

is also available for those with access to the Encodo issue tracker. [Highlights] * Lots of bug fixes and improvements for the Winform UI and German translations with the release of Punchclock ... ]]>

is also available for those with access to the Encodo issue tracker. [Highlights] * Lots of bug fixes and improvements for the Winform UI and German translations with the release of Punchclock on this version. ("QNO-5162" , "QNO-5159" , "QNO-5158" , "QNO-5157" , "QNO-5156" , "QNO-5140" , "QNO-5155" , "QNO-5145" , "QNO-5111" , "QNO-5107" , "QNO-5106" , "QNO-5104" , "QNO-5015" ) * DateTimeExtensions.GetDayOfWeek() had a leap-day bug ("QNO-5051" ) * Fixed how the hash code for GenericObjects is calculated, which fixes sorting issues in grids, specifically for non-persisted or transient objects ("QNO-5137" ) * Improvements to the IAccessControl API for getting groups and users and testing membership ("QNO-5133" ) * Add support for query aliases (e.g. for joining the same table multiple times) ("QNO-531" ) This changes the API surface only minimally. Applications can pass an alias when calling the Join method, as shown below, query.Join(Metadata.Project.Deputy, alias: "deputy") You can find more examples of aliased queries in the TestAliasedQuery(), TestJoinAliasedTables(), TestJoinChildTwice() defined in the QueryTests testing fixture. * Add a standalone IQueryAnalyzer for optimizations and in-memory mini-drivers ("QNO-4830" ) [Breaking changes] * ISchemaManager has been removed. Instead, you should retrieve the interface you were looking for from the IOC. The possible interfaces you might need are IImportHandler, IMappingBuilder, IPlanBuilder or ISchemaCommandFactory. * ISchemaManagerSettings.GetAuthorized() has been moved to ISchemaManagerAuthorizer. * The hash-code fix for GenericObjects may have an effect on the way your application sorts objects.The IParticipantManager (base interface of IAccessControl) no longer has a single method called GetGroups(IParticipant). This method was previously used to get the groups to which a user belongs and the child groups of a given group. This confusing double duty for the API led to an incorrect implementation for both usages. Instead, there are now two methods: * IEnumerable GetGroups(IUser user): Gets the groups for the given user * IEnumerable GetChildGroups(IGroup group): Gets the child groups for the given group The old method has been removed from the interface because (A) it never worked correctly anyway and (B) it conflicts with the new API. ]]> https://www.earthli.com/news/view_article.php?id=3234 2016-03-25T13:41:30+01:00 https://earthli.com/users/marco

. ------------------------------------------------------------------------ This first-ever "Voxxed Zürich" was hosted at the cinema in the SihlCity shopping center in Zürich on March 3rd. All presentations were in English. The conference was relatively small -- 333 participants -- and largely vendor-free. The... ]]>

. -------------------------------------------------------------------------------- This first-ever "Voxxed Zürich" was hosted at the cinema in the SihlCity shopping center in Zürich on March 3rd. All presentations were in English. The conference was relatively small -- 333 participants -- and largely vendor-free. The overal technical level of the presentations and participants was quite high. I had a really nice time and enjoyed a lot of the presentations. There was a nice common thread running through all of the presentations, starting with the Keynote. There's a focus on performance and reliability through immutabiliy, sequences, events, actors, delayed execution (lambdas, which are relatively new to Java), instances in the cloud, etc. It sounds very BUZZWORDY, but instead it came as a very technically polished conference that reminded me of how many good developers there are trying to do the right thing. Looking forward to next year; hopefully Encodo can submit a presentation. You can take a look at the "VoxxedDays Zürich -- Schedule" . The talks that I visited are included below, with links to the presentation page, the video on YouTube and my notes and impressions. YMMV. [Keynote: Life beyond the Illusion of the Present] "Life beyond the Illusion of the Present" -- Jonas Bonér [media] [Notes] * He strongly recommended reading "The Network is reliable" by Peter Bailis. * This talk is about event-driven, CQRS programming. * Focus on immutable state, very much like JoeDuffy, etc. transactional accrual of facts. * Never delete data, annotate with more facts. * The reality at any point can be calculated for a point in time by aggregating facts up to that point. Like the talk I once wrote up some notes about ("Runaway Complexity in Big Data, and a Plan to Stop It" by Nathan Marz ). * Everything else is a performance optimization. Database views, tables are all caches on the transaction log. Stop throwing the log away, though. * Define smaller atomic units. Not a whole database. Smaller. Consistency boundary. Services? * Availability trumps consistency. Use causal consistency through mechanisms other than time stamps. Local partial better than global. * He talked about data-flow programming; fingers crossed that we get some language support in C# 7 * Akka (Akka.NET) is the main product. [Kotlin - Ready for production] "Kotlin - Ready for production" -- Hadi Hariri [media] * Used at JetBrains, open-source. 14k+ users. It's not a ground-breaking language. They tried Scala and Scala was the first language they tried to use (Java already being off the table) but they didn't like it, so they invented Kotlin. * Interoperable with Java (of course). Usable from all sorts of systems, but intelliJ Idea has first-class support. * Much less code, less maintenance. Encapsulates some concepts like "data classes" which do what they're supposed for DTO definitions. * Inferred type on declarations. No nulls. Null-safe by design. Opt-in for nulls. * Implicit casts as well * Interface delegation * Lazy delegation * Deconstruction * Global infix operators; very expressive * Also defaults to/focuses on immutability * Algebraic data types/ data flow * Anglo is statically typed XML views for android * JavaScript target exists and is the focus of work. Replacement for TypeScript? [Reactive Apps with Akka and AngularJS] "Reactive Apps with Akka and AngularJS" -- Heiko Seeberger [media] * He strongly recommended reading "the reactive manifesto" * Responsive: timely response / non-functional / also under load / scale up/down/out * Resilient: fail early * Message-driven: async message-passing is a way of getting reactive/responsive. Automatic decoupling leads to better error-handling, no data loss * Akka provides support for: * Actor-based model (actors are services); watch video from Channel Nine * Akka HTTP Server is relatively new * Akka is written in Scala * There's a Scala DSL for defining the controller (define routes) * The Scala compiler is pure crap. Sooooo slooooowww (62 seconds for 12 files) During his talk, he took us through the following stages of building a scalable, resilient actor-based application with Akka. * First he started with static HTML * Then he moved on to something connected to AKKA, but not refreshing * W3C Server-sent events is unidirectional channel from the server to the client. He next used this to have instant refresh on the client; not available on IE. Probably used by SignalR (or whatever replaced it)? Nothing is typed, though, just plain old JavaScript * Then he set up sharding * Then persistence (Cassandra, Kafka) AKKA Distributed Data * Deals with keeping replicas consistent without central coordination * Conflict-free replicated data types * Fully distributed, has pub/sub semantics * Uses the Gossip protocol * Support various consistency strategies * Using AKKA gives you automated scaling support (unlike the SignalR demo Urs and I did over 2 years ago, but that was a chat app as well) AKKA Cluster Sharding * Partitioning of actors/services across clusters * Supports various strategies * Default strategy is to distribute unbalanced actors to new shards * The ShardRegion is another actor that manages communication with sharded actors (entities). This introduces a new level of indirection, which must be honored in the code (?) AKKA Persistence * Event-sourcing: validate commands, journal events, apply the event after persistence. * Application is applied to local state only after the journal/persistence has indicated that the command was journaled * On recovery, events are replayed * Supports snapshotting (caching points in time) * Requires a change to the actor/entity to use it. All written in Scala. Akka looks pretty good. It guarantees the ordering because ACTORS. Any given actor only exists on any shard once. If a shard goes down, the actor is recreated on a different shard, and filled with information from the persistent store to "recreate" the state of that actor. DDD (Domain-Driven Design) and the actor model. Watch "Hewitt, Meijer and Szyperski: The Actor Model (everything you wanted to know, but were afraid to ask)" . Code is on GitHub: seeberger/reactive_flows [Lambda core - hardcore] "Lambda core - hardcore" -- Jarek Ratajski [media] Focus on immutability and no side-effects. Enforced by the lambda calculus. Pretty low-level talk about lambda calculus. Interesting, but not applicable. He admitted as much at the top of the talk. Links: * "Dixin's Blog" * "Badlam visualization" * "His presentation engine" [expect("poo").length.toBe(1)] "expect("poo").length.toBe(1)" -- Philip Hofstetter [1] [media] This was a talk about expectations of the length of a character. The presenter was very passionate about his talk and went into an incredible amount of detail. * What is a string? This is the kind of stuff every programmer needs to know. [2] * String is not a collection of bytes. It's a sequence of graphemes. string <> char[] * UTF-16 is crap. What about the in-memory representation? Why in God's name did Python 3 use UTF32? Unicode Transformation format. * What is the length of a string? ä is how many? Single character (diuresis included) or a with combining diuresis? * In-memory representation in Java and C# are UCS-2 (UNICODE 1); stuck in 1996, before Unicode 2.0 came out. This leaks into APIs because of how strings are returned ... string APIs use UTF-16, encoding with surrogate pairs to get to characters outside of the BMP (understood by convention, but not by the APIs that expect UTF-16 ... which has no idea what surrogate pairs are ... and counting algorithms, find, etc. won't work). * ECMAScript hasn't really fixed this, either. substr() can break strings charAt() is still available and has no idea about code points. Does this apply to ES6? String-equality doesn't work for the diuresis above. * So we're stuck with server-side. Who does it right? Perl. Swift. Python. Ruby. Python went through hell with backwards compatibility but with 3.3 they're doing OK again. Ruby strings are a tuple of encoding and data. All of the others have their string libraries dealing in graphemes. How did Perl always get it right? Perl has three methods for asking questions about length, in graphemes, code points or bytes * What about those of us using JavaScript? C#? Java? There are external libraries that we should be using. Not just for DateTime, but for string-handling as well. Even EcmaScript15 still uses code points rather than graphemes, so the count varies depending on how the grapheme is constructed. * Security concerns: certificate authorities have to be aware of homographs (e.g. a character that looks like another one, but has a different encoding/byte sequence). * He recommended the book "Unicode explained" by Jukka K. Korpela. [How usability fits in UX - it's no PICNIC] "How usability fits in UX - it's no PICNIC" -- Myriam Jessier [media] What should a UI be? 1. Functional 2. Reliable 3. Usable 4. Convenient 5. Pleasurable Also nice to have: 1. Desirable 2. Delightful 3. memorable 4. Learnable 5. 3 more Book recommendation: "Don't make me think" by Steve Krug * Avoid mindless and unambiguous clicks. Don't count clicks, count useless shit you need to do. * Let the words go. People's attention will wander. * UX is going to be somewhat subjective. Don't try to please everyone. * OMG She uses hyphens correctly. * She discussed the difference between UX, CX, UI. * Personas are placeholders for your users. See "Personapp" to get started working with personas. Guidelines: * Consistent and standardized UI * Guide the user (use visual cues, nudging) * Make the CallToAction (CTA) interactive objects obvious * Give feedback on progress, interaction * Never make a user repeat something they already told you. You're software, you should have eidetic memory * Always have default values in forms (e.g. show the expected format) * Explain how the inputed information will be used (e.g. for marketing purposes) * No more "reset" button or mass-delete buttons. Don't make it possible/easy to wipe out all someone's data * Have clear and explanatory error or success messages (be encouraging) * Include a clear and visual hierarchy and navigation Guidelines for mobile: * Make sure it works on all phones * Give incentives for sharing and purpose (engagement rates make marketing happy. CLICK THE BUTTON) * Keep usability and conversion in mind (not necessarily money, but you actually want people to be using your app correctly) * Usability (can you use your app on the lowest screen-brightness?) * ...and more... * Make it pretty (some people don't care, e.g. She very clearly said that she's not aesthetically driven, it's not her field; other people do care. A lot). * Give all the information a customer needs to purchase * Design for quick movement (no lag) * Do usability testing through video * Leverage expectations. Fit in to the environment. Search is on the left? Behind a button? Do that. Don't make a new way of searching. * If you offer a choice, then make them as mutually exclusive as possible. When a company talks to itself (e.g. industry jargon), then users get confused * The registration process should be commensurate to the thing that you're registering for * Small clickable ads on mobile. Make click targets appropriate. * Don't blame negative feedback on "fear of change". It's probably you. If people don't like it, then it might not be user-friendly. The example with Twitter's star vs. heart. It's interesting how we let the world frame our interactions. Why not both? Too complex? Would people really be confused by two buttons? One to "like" and one for "read later"? Suggested usability testing tools: * Crazy Egg is $9 per month for heatmaps. * Qualaroo * Optimizely (A/B testing) * Usabilia * Userfeel * Trymyui [React - A trip to Russia isn't all it seems] "React - A trip to Russia isn't all it seems" -- Josh Sephton [3] [media] This talk was about Web UI frameworks and how his team settled on React. * Angular too "all or nothing". * Backbone has no data-binding. * React looks good. Has its own routing for SPAs. Very component-heavy. Everything's a component. Nothing new here so far. * They built their React to replace a Wordpress-based administration form * Stateful components are a bad idea * React components are like self-contained actors/services * They started with Flux, but ended up with Redux. We're using Redux in our samples. I'm eyeballing how to integrate Akka.Net (although I'm not sure if that has anything to do with this. * ReactNative: write once, use on any device * Kind of superficial and kinda short but I knew all about this in React already [The reactor programming model for composable distributed computing] "The reactor programming model for composable distributed computing" -- Aleksandar Prokopec [4] [media] * Reactive programming, with events as sequences of event objects * Events are equivalent to a list/sequence/streams (enumerable in C#) * This talk is also about managing concurrency * There must be a boundary between outer concurrent events vs. how your application works on them * That's why most UI toolkits are single-threaded * Asynchronous is the antonym of concurrency (at least in the dictionary) * Filter the stream of events to compress them to frames, then render and log, so the events come in, are marshaled through the serializing bottleneck and are then dispatched asynchronously to different tasks * Reactor lets clients create their own channels (actors) from which they read events and which they register with a server so that it can publish * Akka supports setting up these things, Reactor is another implementation? * Dammit I want destructuring of function results (C# 7?) * It's very easy to build client/server and broadcast and even ordered synchronization using UIDs (or that pattern mentioned by Jonas in the keynote) The UID needs to be location-specific, though. That's not sufficient either, what you need is client-specific. For this, you need special data structures to store the data in a way that edits are automatically correctly ordered. Events sent for these changes make the events are ordered correctly * What is the CRDT? We just implemented an online collaborative editor: composes nicely and provides a very declarative, safe and scalable way of defining software. This is just a function (feeds back into the idea of lambdas here, actually, immutability, encapsulation) * "Reactors" -------------------------------------------------------------------------------- [1] I am aware of the irony that the emoji symbol for "poo" is not supported on this blogging software. That was basically the point of the presentation -- that encoding support is difficult to get right. There's an issue for it: "Add support for UTF8 as the default encoding" . [1] In my near-constant striving to be the worst conversational partner ever, I once gave a similar encoding lesson to my wife on a two-hour walk around a lake when she dared ask why mails sometimes have those "stupid characters" in them. ]]> https://www.earthli.com/news/view_article.php?id=3233 2016-03-25T13:39:04+01:00 https://earthli.com/users/marco

"[image]" At the beginning of the year, we worked on an interesting project that dipped into IOT (Internet of Things). The project was to create use cases for Crealogix's banking APIs in the real world. Concretely, we wanted to show how a customer could use these APIs in their own workflows. The use cases... ]]>

"[image]" At the beginning of the year, we worked on an interesting project that dipped into IOT (Internet of Things). The project was to create use cases for Crealogix's banking APIs in the real world. Concretely, we wanted to show how a customer could use these APIs in their own workflows. The use cases were to provide proof of the promise of flexibility and integrability offered by well-designed APIs. "Watch 7--minute video of the presentation" [The Use Cases] [Football Club Treasurer] [image]The first use case is for the treasurer of a local football club. The treasurer wants to be notified whenever an annual club fee is transferred from a member. The club currently uses a Google Spreadsheet to track everything, but it's updated manually. It would be really nice if the banking API could connected -- via some scripting "glue" -- to update the spreadsheet directly, without user intervention. The treasurer would just see the most current numbers whenever he opened the spreadsheet. The spreadsheet is in addition to the up-to-date view of payments in the banking app. The information is also available there, but not necessarily in the form that he or she would like. Linking automatically to the spreadsheet is the added value. [Chore & Goal Tracker] [image][image]Imagine a family with a young son who wants to buy a drone. He would have to earn it by doing chores. Instead of tracking this manually, the boy's chores would be tabulated automatically, moving money from the parents' account to his own as he did chores. Additionally, a lamp in the boy's room would glow a color indicating how close he was to his goal. The parents wanted to track the boy's progress in a spreadsheet, tracking the transfers as they would have had they not had any APIs. The idea is to provided added value to the boy, who can record his chores by pressing a button and see his progress by looking at a lamp's color. The parents get to stay in their comfort zone, working with a spreadsheet as usual, but having the data automatically entered in the spreadsheet. [The Plan] It's a bit of a stretch, but it sufficed to ground the relatively abstract concept of banking APIs in an example that non-technical people could follow. So we needed to pull quite a few things together to implement these scenarios. * A lamp that can be controlled via API * A button that can trigger an API * A spreadsheet accessibly via API * An API that can transfer money between accounts * "Glue" logic that binds these APIs together [The Lamp] [image][image] We looked at two lamps: * "Philips Hue" * "Lifx" Either of these -- just judging from their websites -- would be sufficient to utterly and completely change our lives. The Hue looked like it was going to turn us into musicians, so we went with Lifx, which only threatened to give us horn-rimmed glasses and a beard (and probably skinny jeans and Chuck Taylor knockoffs). Yeah, we think the marketing for what is, essentially, a light-bulb, is just a touch overblown. Still, you can change the color of the light bulb with a SmartPhone app, or control it via API (which is what we wanted to do). [The Button] The button sounds simple. You'd think that, in 2016, these things would be as ubiquitous as AOL CDs were in the 1990s. You'd be wrong. [image]There's a KickStarter project called "Flic" that purports to have buttons that send signals over a wireless connection. They cost about CHF20. Though we ordered some, we never saw any because of manufacturing problems. If you thought the hype and marketing for a light bulb were overblown, then you're sure to enjoy how Flic presents a button. We quickly moved along a parallel track to get buttons that can be pressed in real life rather than just viewed from several different angles and in several different colors online. [image]Amazon has what they have called "Dash" buttons that customers can press to add predefined orders to their one-click shopping lists. The buttons are bound to certain household products that you tend to purchase cyclically: toilet paper, baby wipes, etc. They sell them dirt-cheap -- $5 -- but only to Amazon Prime customers -- and only to customers in the U.S. Luckily, we knew someone in the States willing to let us use his Amazon Prime account to deliver them, naturally only to a domestic address, from which they would have to be forwarded to us here in Switzerland. That we couldn't use them to order toilet paper in the States didn't bother us -- we were planning to hack them anyway. These buttons showed up after a long journey and we started trapping them in our own mini-network so that we could capture the signal they send and interpret it as a trigger. This was not ground-breaking stuff, but we really wanted the demonstrator to be able to press a physical button on stage to trigger the API that would cascade other APIs and so on. Of course we could have just hacked the whole thing so that someone presses a button on a screen somewhere -- and we programmed this as a backup plan -- but the physicality of pressing a button was the part of the demonstration that was intended to ground the whole idea for non-technical users. [1] [The Spreadsheet] [image][image]If you're going to use an API to modify a spreadsheet, then that spreadsheet has to be available online somewhere. The spreadsheet application in Google Docs is a good candidate. The API allows you to add or modify existing data, but that's pretty much it. When you make changes, they show up immediately, with no ceremony. That, unfortunately, doesn't make for a very nice-looking demo. Google Docs also offers a Javascript-like scripting language that let's you do more. We wanted to not only insert rows, we wanted charts to automatically update and move down the page to accommodate the new row. All animated, thank you very much. This took a couple pages of scripting and a good amount of time. It's also no longer a solution that an everyday user is likely to make themselves. And, even though we pushed as hard as we could, we also didn't get everything we wanted. The animation is very jerky (watch the video linked above) but gets the job done. [The Glue] [image]So we've got a bunch of pieces that are all capable of communicating in very similar ways. The final step is to glue everything together with a bit of script. There are several services available online, like IFTTT -- If This Then That -- that allow you to code simple logic to connect signals to actions. In our system, we had the following signals: * Transfer was made to a bank account * Button was pressed and the following actions: * Insert data into Google Spreadsheet * Set color of lamp [The Crealogix API and UI] [image][image][image]So we're going to betray a tiny secret here. Although the product demonstrated on-stage did actually do what it said, it didn't do it using the Crealogix API to actually transfer money. That's the part that we were actually selling and it's the part we ended up faking/mocking out because the actual transfer is beside the point. Setting up bank accounts is not so easy, and the banks take umbrage at creating them for fake purposes. Crealogix could have let us use fake testing accounts, but even that would have been more work than it was worth: if we're already faking, why not just fake in the easiest way possible by skipping the API call to Crealogix and only updating the spreadsheet? Likewise, the entire UI that we included in the product was mocked up to include only the functionality required by the demonstration. You can see an example here -- of the login screen -- but other screens are linked throughout this article. Likewise, the Bank2Things screen shown above and to the left is a mockup. [Wrapup] So what did Encodo actually contribute? * We used the Crealogix UX and VSG to mock up all of the app screens that you seen linked in this article. We did all of the animation and logic and styling. * We built two Google Spreadsheets and hooked them up to everything else * We hooked up the Lifx lamp API into our system * We hacked the Amazon Dash buttons to communicate in our own network instead of beaming home to the mothership * We built a web site to handle any mocking/faking that needed to be done for the demo and through which the devices communicated * We provided a VM (Virtual Machine) on which everything ran (other than the Google Spreadsheets) As last year -- when we helped Crealogix create the prototype for their "BankClip for Finovate 2015" -- we had a lot of fun investigating all of these cutting-edge technologies and putting together a custom solution in time for Finovate 2016. -------------------------------------------------------------------------------- [1] As it turns out, if you watch the "7--minute video of the presentation" , nowhere do you actually see a button. Maybe they could see them from the audience. ]]> https://www.earthli.com/news/view_article.php?id=3222 2016-02-27T12:36:39+01:00 https://earthli.com/users/marco

result = app.GetInstance() ); if (transcript.ExitCode != ExitCodes.Ok) { throw new InvalidOperationException( "Could not read the service settings from the configuration file." + new SimpleMessageFormatter().GetErrorDetails(transcript.Messages) ); } return result; } If you've been following along in the other articles (see first footnote below), then this structure should be very familiar. We use an ApplicationManager() to execute the application logic, creating the application with CreateServiceConfigurationApplication and returning the settings configured by the application in the second parameter (the "run" action). If anything went wrong, we get the details and throw an exception. You can't see it, but the library provides debug/file logging (if you enable it), debug/release mode support (exception-handling, etc.) and everything is customizable/replaceable by registering with an IOC. [Configuring the Settings Loader] Soooo...I can see where we're returning the ServiceSettings, but where are they configured? Let's take a look at the second method, the one that creates the application. private static IApplication CreateServiceConfigurationApplication() { var application = new Application(); application .UseSimpleInjector() .UseStandard() .UseConfigurationFile("service-settings.xml") .Configure( "service", (settings, node) => { settings.ServiceName = node.GetValue("name", settings.ServiceName); settings.DisplayName = node.GetValue("displayName", settings.DisplayName); settings.Description = node.GetValue("description", settings.Description); settings.Types = node.GetValue("types", settings.Types); } ).RegisterSingle(); return application; } 1. First, we create a standard Application, defined in the Encodo.Application assembly. What does this class do? It does very little other than manage the main IOC (see articles linked in the first footnote for details). 2. The next step is to choose an IOC, which we do by calling UseSimpleInjector(). Quino includes support for the SimpleInjector IOC out of the box. As you can see, you must include this support explicitly, so you're also free to assign your own IOC (e.g. one using Microsoft's Unity). SimpleInjector is very lightweight and super-fast, so there's no downside to using it. 3. Now we have an application with an IOC that doesn't have any registrations on it. How do we get more functionality? By calling methods like UseStandard(), defined in the Encodo.Application.Standard assembly. Since I know that UseStandard() pulls in what I'm likely to need, I'll just use that. [4] 4. The next line tells the application the name of the configuration file to use. [5] 5. The very next line is already application-specific code, where we configure the ServiceSettings object that we want to return. For that, there's a Configure method that returns an object from the IOC along with a specific node from the configuration data. This method is called only if everything started up OK. 6. The final call to RegisterSingle makes sure that the ServiceSettings object created by the IOC is a singleton (it would be silly to configure one instance and return another, unconfigured one). Basically, because this application is so simple, it has already accomplished its goal by the time the standard startup completes. At the point that we would "run" this application, the ServiceSettings object is already configured and ready for use. That's why, in LoadServiceSettings(), we can just get the settings from the application with GetInstance() and exit immediately. [Code Generator] The code generator has a bit more code, but follows the same pattern as the simple application above. In this case, we use the command line rather than the configuration file to get user input. [Execution] The main method defers all functionality to the ApplicationManager, passing along two methods, one to create the application, the other to run it. internal static void Main() { new ApplicationManager().Run(CreateApplication, GenerateCode); } [Configuration] As before, we first create an Application, then choose the SimpleInjector and some standard configuration and registrations with UseStandard(), UseMetaStandardServices() and UseMetaTools(). [6] We set the application title to "Quino Code Generator" and then include objects with UseSingle() that will be configured from the command line and used later in the application. [7] And, finally, we add our own ICommandSet to the command-line processor that will configure the input and output settings. We'll take a look at that part next. private static IApplication CreateApplication( IApplicationCreationSettings applicationCreationSettings) { var application = new Application(); return application .UseSimpleInjector() .UseStandard() .UseMetaStandardServices() .UseMetaTools() .UseTitle("Quino Code Generator") .UseSingle(new CodeGeneratorInputSettings()) .UseSingle(new CodeGeneratorOutputSettings()) .UseUnattendedCommand() .UseCommandSet(CreateGenerateCodeCommandSet(application)) .UseConsole(); } [Command-line Processing] The final bit of the application configuration is to see how to add items to the command-line processor. Basically, each command set consists of required values, optional values and zero or more switches that are considered part of a set. The one for i simply sets the value of inputSettings.AssemblyFilename to whatever was passed on the command line after that parameter. Note that it pulls the inputSettings from the application to make sure that it sets the values on the same singleton reference as will be used in the rest of the application. The code below shows only one of the code-generator--specific command-line options. [8] private static ICommandSet CreateGenerateCodeCommandSet( IApplication application) { var inputSettings = application.GetSingle(); var outputSettings = application.GetSingle(); return new CommandSet("Generate Code") { Required = { new OptionCommandDefinition { ShortName = "i", LongName = "in", Description = Resources.Program_ParseCommandLineArgumentIn, Action = value => inputSettings.AssemblyFilename = value }, // And others... }, }; } [Code-generation] Finally, let's take a look at the main program execution for the code generator. It shouldn't surprise you too much to see that the logic consists mostly of getting objects from the IOC and telling them to do stuff with each other. [9] I've highlighted the code-generator--specific objects in the code below. All other objects are standard library tools and interfaces. private static void GenerateCode(IApplication application) { var logger = application.GetLogger(); var inputSettings = application.GetInstance(); if (!inputSettings.TypeNames.Any()) { logger.Log(Levels.Warning, "No types to generate."); } else { var modelLoader = application.GetInstance(); var metaCodeGenerator = application.GetInstance(); var outputSettings = application.GetInstance(); var modelAssembly = AssemblyTools.LoadAssembly( inputSettings.AssemblyFilename, logger ); outputSettings.AssemblyDetails = modelAssembly.GetDetails(); foreach (var typeName in inputSettings.TypeNames) { metaCodeGenerator.GenerateCode( modelLoader.LoadModel(modelAssembly, typeName), outputSettings, logger ); } } } So that's basically it: no matter how simple or complex your application, you configure it by indicating what stuff you want to use, then use all of that stuff once the application has successfully started. The Encodo/Quino framework provides a large amount of standard functionality. It's yours to use as you like and you don't have to worry about building it yourself. Even your tiniest application can benefit from sophisticated error-handling, command-line support, configuration and logging without lifting a finger. -------------------------------------------------------------------------------- [1] See Encodo’s configuration library for Quino "Part 1" , "Part 2" and "Part 3" as well as API Design: Running and Application "Part 1" and "Part 2" and, finally, "Starting up an application, in detail" . [1] That boilerplate looks like this: var fileService = new ServiceInstaller(); fileService.StartType = ServiceStartMode.Automatic; fileService.DisplayName = "Quino Sandbox"; fileService.Description = "Demonstrates a Quino-based service."; fileService.ServiceName = "Sandbox.Services"; See the ServiceInstaller.cs file in the Sandbox.Server project in Quino 2.1.2 and higher for the full listing. [1] The standard implementation of Quino's ITextKeyValueNodeReader supports XML, but it would be trivial to create and register a version that supports JSON ("QNO-4993" ) or YAML. The configuration file for the utility looks like this: Quino.Services Quino Utility The application to run all Quino backend services. All [1] If you look at the implementation of the UseStandard method [10], it pulls in a lot of stuff, like support for BCrypt, enhanced CSV and enum-value parsing and standard configuration for various components (e.g. the file log and command line). It's called "Standard" because it's the stuff we tend to use in a lot of applications. But that method is just a composition of over a dozen other methods. If, for whatever reason (perhaps dependencies), you don't want all of that functionality, you can just call the subset of methods that you do want. For example, you could call UseApplication() from the Encodo.Application assembly instead. That method includes only the support for: * Processing the command line (ICommandSetManager) * Locating external files (ILocationManager) * Loading configuration data from file (IConfigurationDataLoader) * Debug- and file-based logging (IExternalLoggerFactory) * and interacting with the IApplicationManager. If you want to go even lower than that, you can try UseCore(), defined in the Encodo.Core assembly and then pick and choose the individual components yourself. Methods like UseApplication() and UseStandard() are tried and tested defaults, but you're free to configure your application however you want, pulling from the rich trove of features that Quino offers. [1] By default, the application will look for this file next to the executable. You can configure this as well, by getting the location manager with GetLocationManager() and setting values on it. You'll notice that I didn't use Configure() for this particular usage. That's ordinarily the way to go if you want to make changes to a singleton before it is used. However, if you want to change where the application looks for configuration files, then you have to change the location manager before it's used any other configuration takes place. It's a special object that is available before the IOC has been fully configured. To reiterate from other articles (because it's important), the order of operations we're interested in here are: 1. Create application (this is where you call Use*() to build the application) 2. Get the location manager to figure out the path for LocationNames.Configuration 3. Load the configuration file 4. Execute all remaining actions, including those scheduled with calls to Configure() If you want to change the configuration-file location, then you have to get in there before the startup starts running -- and that's basically during application construction. Alternatively, you could also call UseConfigurationDataLoader() to register your own object to actually load configuration data and do whatever the heck you like in there, including returning constant data. :-) [1] The metadata-specific analog to UseStandard() is UseMetaStandard(), but we don't call that. Instead, we call UseMetaStandardServices(). Why? The answer is that we want the code generator to be able to use some objects defined in Quino, but the code generator itself isn't a metadata-based application. We want to include the IOC registrations required by metadata-based applications without adding any of the startup or shutdown actions. Many of the standard Use*() methods included in the base libraries have analogs like this. The Use*Services() analogs are also very useful in automated tests, where you want to be able to create objects but don't want to add anything to the startup. [1] Wait, why didn't we call RegisterSingle()? For almost any object, we could totally do that. But objects used during the first stage of application startup -- before the IOC is available -- must go in the other IOC, accessed with SetSingle() and GetSingle(). [1] The full listing is in Program.cs in the Quino.CodeGenerator project in any 2.x version of Quino. [1] Note that, once the application is started, you can use GetInstance() instead of GetSingle() because the IOC is now available and all singletons are mirrored from the startup IOC to the main IOC. In fact, once the application is started, it's recommended to use GetInstance() everywhere, for consistency and to prevent the subtle distinction between IOCs -- present only in the first stage of startup -- from bleeding into your main program logic. [1] If you have the Quino source code handy, you can look it up there, but if you have ReSharper installed, you can just F12 on UseStandard() to decompile the method. In the latest DotPeek, the extension methods are even displayed much more nicely in decompiled form. ]]> https://www.earthli.com/news/view_article.php?id=3207 2016-01-17T22:27:10+01:00 https://earthli.com/users/marco

provides a typically twisted and unhelpful overview of the state of functional programming in this 21st-century renaissance -- heralded decades ago by Lisp programmers. It includes an honest overview of the major players, including Scala, for... ]]>

provides a typically twisted and unhelpful overview of the state of functional programming in this 21st-century renaissance -- heralded decades ago by Lisp programmers. It includes an honest overview of the major players, including Scala, for which the "pro" and "con" are the same (a "[c]lose relationship with Java [...]") and ending with JavaScript, for which the "pro" is "It's what you'll end up using." The discussion continues with rules: variable immutability, function purity, curryability and monadicity, which is where things really go off the rails. Property 7 dribbles to a shuddering halt with, "All monads define a unit() function called of(), a bind() function called map() and a type constructor function called... "Wait a minute. Wait a minute. Perhaps bind() is a functor not a function. I'm pretty sure about that. Hold on to the horses a moment there while I look it up. "...And I should perhaps clarify that this bind() and map() is nothing to do with any other bind() or map() methods or functions that you might be familiar with, although their actions are in some sense quite similar. "Summary: It has been an honour and a pleasure to clear all that up for you. "Final Reader's comment: My gratitude is inexpressible. [1]" -------------------------------------------------------------------------------- [1] I've been reading Verity Stob since the days when she was published in Dr. Dobbs. Nice to see her still going with the same sense of humour (sic) and insight into the at-times puffed-up and self-important programming world. [2] [1] Which is not to say that I don't enjoy immensely the functional aspects of C#. I do. I also have read a lot about monads and am completely familiar with the tragically bad and unenlightening explanations. Stob captures this elegantly with the following corollary to Rule 4: "If you should by some accident come to understand what a Monad is, you will simultaneously lose the ability to explain it to anybody else." ]]> https://www.earthli.com/news/view_article.php?id=3204 2016-01-16T12:53:04+01:00 https://earthli.com/users/marco

of this series, we discussed some core concepts of profiling. In that article, we not only discussed the problem at hand, but also how to think about not only fixing performance problems, but reducing the likelihood that they get out of hand in the first place. In this second part, we'll... ]]>

of this series, we discussed some core concepts of profiling. In that article, we not only discussed the problem at hand, but also how to think about not only fixing performance problems, but reducing the likelihood that they get out of hand in the first place. In this second part, we'll go into detail and try to fix the problem. [Reëvaluating the Requirements] Since we have new requirements for an existing component, it's time to reconsider the requirements for all stakeholders. In terms of requirements, the IScope can be described as follows: 1. Hold a list of objects in LIFO order 2. Hold a list of key/value pairs with a unique name as the key 3. Return the value/reference for a key 4. Return the most appropriate reference for a given requested type. The most appropriate object is the one that was added with exactly the requested type. If no such object was added, then the first object that conforms to the requested type is returned 5. These two piles of objects are entirely separate: if an object is added by name, we do not expect it to be returned when a request for an object of a certain type is made There is more detail, but that should give you enough information to understand the code examples that follow. [Usage Patterns] There are many ways of implementing the functional requirements listed above. While you can implement the feature with only requirements, it's very helpful to know usage patterns when trying to optimize code. Therefore, we'd like to know exactly what kind of contract our code has to implement -- and to not implement any more than was promised. Sometimes a hopeless optimization task gets a lot easier when you realize that you only have to optimize for a very specific situation. In that case, you can leave the majority of the code alone and optimize a single path through the code to speed up 95% of the calls. All other calls, while perhaps a bit slow, will at least still be yield the correct results. And "optimized" doesn't necessarily mean that you have to throw all of your language's higher-level constructs out the window. Once your profiling tool tells you that a particular bit of code has introduced a bottleneck, it often suffices to just examine that particular bit of code more closely. Just picking the low-hanging fruit will usually be more than enough to fix the bottleneck. [1] [Create scopes faster [2]] I saw in the profiler that creating the ExpressionContext had gotten considerably slower. Here's the code in the constructor. foreach (var value in values.Where(v => v != null)) { Add(value); } I saw a few potential problems immediately. * The call to Add() had gotten more expensive in order to return the most appropriate object from the GetInstances() method * The Linq replaced a call to AddRange() The faster version is below: var scope = CurrentScope; for (var i = 0; i < values.Length; i++) { var value = values[i]; if (value != null) { scope.AddUnnamed(value); } } Why is this version faster? The code now uses the fact that we know we're dealing with an indexable list to avoid allocating an enumerator and to use non-allocating means of checking null. While the Linq code is highly optimized, a for loop is always going to be faster because it's guaranteed not to allocate anything. Furthermore, we now call AddUnnamed() to use the faster registration method because the more involved method is never needed for these objects. The optimized version is less elegant and harder to read, but it's not terrible. Still, you should use these techniques only if you can prove that they're worth it. [Optimizing CurrentScope] Another minor improvement is that the call to retrieve the scope is made only once regardless of how many objects are added. On the one hand, we might expect only a minor improvement since we noted above that most use cases only ever add one object anyway. On the other, however, we know that we call the constructor 20 million times in at least one test, so it's worth examining. The call to CurrentScope gets the last element of the list of scopes. Even something as innocuous as calling the Linq extension method Last() can get more costly than it needs to be when your application calls it millions of times. Of course, Microsoft has decorated its Linq calls with all sorts of compiler hints for inlining and, of course, if you decompile, you can see that the method itself is implemented to check whether the target of the call is a list and use indexing, but it's still slower. There is still an extra stack frame (unless inlined) and there is still a type-check with as. Replacing a call to Last() with getting the item at the index of the last position in the list is not recommended in the general case. However, making that change in a provably performance-critical area shaved a percent or two off a test run that takes about 45 minutes. That's not nothing. protected IScope CurrentScope { get { return _scopes.Last(); } } protected IScope CurrentScope { get { return _scopes[_scopes.Count - 1]; } } That takes care of the creation & registration side, where I noticed a slowdown when creating the millions of ExpressionContext objects needed by the data driver in our product's test suite. [Get objects faster] Let's now look at the evaluation side, where objects are requested from the context. The offending, slow code is below: public IEnumerable GetInstances() { var serviceType = typeof(TService); var rawNameMatch = this[serviceType.FullName]; var memberMatches = All.OfType(); var namedMemberMatches = NamedMembers.Select( item => item.Value ).OfType(); if (rawNameMatch != null) { var nameMatch = (TService)rawNameMatch; return nameMatch .ToSequence() .Union(namedMemberMatches) .Union(memberMatches) .Distinct(ReferenceEqualityComparer.Default); } return namedMemberMatches.Union(memberMatches); } As you can readily see, this code isn't particularly concerned about performance. It is, however, relatively easy to read and to figure out the logic behind returning objects, though. As long as no-one really needs this code to be fast -- if it's not used that often and not used in tight loops -- it doesn't matter. What matters more is legibility and maintainability. But we now know that we need to make it faster, so let's focus on the most-likely use cases. I know the following things: * Almost all Scope instances are created with a single object in them and no other objects are ever added. * Almost all object-retrievals are made on such single-object scopes * Though the scope should be able to return all matching instances, sorted by the rules laid out in the requirements, all existing calls get the FirstOrDefault() object. These extra bits of information will allow me to optimize the already-correct implementation to be much, much faster for the calls that we're likely to make. The optimized version is below: public IEnumerable GetInstances() { var members = _members; if (members == null) { yield break; } if (members.Count == 1) { if (members[0] is TService) { yield return (TService)members[0]; } yield break; } object exactTypeMatch; if (TypedMembers.TryGetValue(typeof(TService), out exactTypeMatch)) { yield return (TService)exactTypeMatch; } foreach (var member in members.OfType()) { if (!ReferenceEquals(member, exactTypeMatch)) { yield return member; } } } Given the requirements, the handful of use cases and decent naming, you should be able to follow what's going on above. The code contains many more escape clauses for common and easily handled conditions, handling them in an allocation-free manner wherever possible. 1. Handle empty case 2. Handle single-element case 3. Return exact match 4. Return all other matches [3] You'll notice that returning a value added by-name is not a requirement and has been dropped. Improving performance by removing code for unneeded requirements is a perfectly legitimate solution. [Test Results] And, finally, how did we do? I created tests for the following use cases: * Create scope with multiple objects * Get all matching objects in an empty scope * Get first object in an empty scope * Get all matching objects in a scope with a single object * Get first object in a scope with a single object * Get all matching objects in a scope with multiple objects * Get first object in a scope with multiple objects Here are the numbers from the automated tests. [image] [image] * Create scope with multiple objects -- 12x faster * Get all matching objects in an empty scope -- almost 2.5x faster * Get first object in an empty scope -- almost 3.5x faster * Get all matching objects in a scope with a single object -- over 3x faster * Get first object in a scope with a single object -- over 3.25x faster * Get all matching objects in a scope with multiple objects -- almost 3x faster * Get first object in a scope with multiple objects -- almost 2.25x faster This looks amazing but remember: while the optimized solution may be faster than the original, all we really know is that we've just managed to claw our way back from the atrocious performance characteristics introduced by a recent change. We expect to see vast improvements versus a really slow version. Since I know that these calls showed up as hotspots and were made millions of times in the test, the performance improvement shown by these tests is enough for me to deploy a pre-release of Quino via TeamCity, upgrade my product to that version and run the tests again. Wish me luck! [4] -------------------------------------------------------------------------------- [1] Or, most likely, push it to some other piece of code. [1] The best approach at this point is to create issues for the other performance investigations you could make. For example, I opened an issue called "Optimize allocations in the data handlers (start with IExpressionContexts)" , documented everything I had analyzed and quickly got back to the issue on which I'd started. [1] For those with access to the Quino Git repository, the diffs shown below come from commit a825d5030ce6f65a452e1db85a308e1351288b96. [1] If you're following along very, very carefully, you'll recall at this point that the requirement stated above is that objects are returned in LIFO order. The faster version of the code returns objects in FIFO order. You can't tell that the original, slow version did guarantee LIFO ordering, but only because the call to get All members contained a hidden call to the Linq call Reverse(), which slowed things down even more! I removed the call to reverse all elements because (A) I don't actually have any tests for the LIFO requirement nor (B) do I have any other code that expects it to happen. I wasn't about to make the code even more complicated and possibly slower just to satisfy a purely theoretical requirement. That's the kind of behavior that got me into this predicament in the first place. [1] Spoiler alert: it worked. ;-) The fixes cut the testing time from about 01:30 to about 01:10 for all tests on the build server, so we won back the lost 25%. ]]> https://www.earthli.com/news/view_article.php?id=3203 2016-01-13T07:05:23+01:00 https://earthli.com/users/marco

, we'll describe the problem in detail and take a crack at improving the speed. See you there. -------------------------------------------------------------------------------- [1] In fairness, the quote is at least properly attributed. It really was Donald Knuth who wrote it. [1] By "opportunity", of course, I mean that I messed something up that made Quino slower in the new version. [1] See the article "Quino 2: Starting up an application, in detail" for more information on this usage. [1] I'm working on this right now, in issue "Add standard performance tests" for release 2.1. ]]> https://www.earthli.com/news/view_article.php?id=3195 2016-01-01T22:52:49+01:00 https://earthli.com/users/marco

is also available for those with access to the Encodo issue tracker. [Highlights] Quino 2 is finally ready and will go out the door with a 2.1 rather than a 2.0 version number. The reason... ]]>

is also available for those with access to the Encodo issue tracker. [Highlights] Quino 2 is finally ready and will go out the door with a 2.1 rather than a 2.0 version number. The reason being that we released 2.0 internally and tested the hell out of it. 2.1 is the result of that testing. It includes a lot of bug fixes as well as API tweaks to make things easier for developers. * Made more improvements to the Startup/Configuration API for products ("QNO-4860" , "QNO-4991" , "QNO-4990" , "QNO-4911" ) * Further refined the independent and properly decoupled assemblies ("QNO-5001" , "QNO-4609" , "QNO-2514" , "QNO-4980" , "QNO-4974" , "QNO-4865" ) * Improved support for continuous integration and deployment of Quino packages and symbols via TeamCity ("QNO-5003" , "QNO-3738" , "QNO-4998" , "QNO-4756" , "QNO-4995" ) * Improved the Web integration with ASP.NET WebAPI and MVC authentication filters ("QNO-4711" ) * Addressed some performance regressions from 1.13 and added a suite of performance tests to keep better track of performance targets. On top of that, I've gone through the backlog and found many issues that had either been fixed already, were obsolete or had been inadequately specified. The Quino backlog dropped from 682 to 542 issues. * 18 issues marked as won't fix and 46 issues marked as obsolete * Stop supporting Glimpse, although there is a Quino.Web.Glimpse package to use the support we do have ("QNO-4560" ) * Stop supporting HtmlHelpers and other client-side rendering ("QNO-3921" , "QNO-3995" , "QNO-3804" , "QNO-3797" , "QNO-3974" , "QNO-4001" , "QNO-3992" , "QNO-3991" , "QNO-3973" , "QNO-3970" , "QNO-3969" , "QNO-3918" , "QNO-3866" , "QNO-3865" , "QNO-3857" , "QNO-3849" , "QNO-3848" , "QNO-3842" , "QNO-3839" , "QNO-3837" , "QNO-3836" , "QNO-3834" , "QNO-3833" , "QNO-3831" , "QNO-3824" w/sub-tasks, "QNO-3806" , "QNO-3805" , "QNO-3802" , "QNO-2288" ) 12 issues marked as incomplete (not sufficiently specified) * 2 issues marked as cannot reproduce * 97 issues marked as fixed [Breaking changes] The following changes are marked with Obsolete attributes, so you'll get a hint as to how to fix the problem. Since these are changes from an unreleased version of Quino, they cause a compile error. * UseMetaSchemaWinformDxFeedback() has been renamed to UseMetaschemaWinformDx() * UseSchemaMigrationSupport() has been renamed to UseIntegratedSchemaMigration() * MetaHttpApplicationBase.MetaApplication has been renamed to BaseApplication * The IServer.Run() extension method is no longer supported. * GetStandardFilters, GetStandardFiltersForFormsAuthentication() and GetStandardFiltersForUnrestrictedAuthentication are no longer supported. Instead, you should register filters in the IOC and use the IWebFilterAttributeFactory.CreateFilters() to get the list of supported filters * The ToolRequirementAttribute is no longer supported or used. * AssemblyExtensions.GetLoadableTypesWithInterface() is no longer supported * AssemblyTools.GetValidAssembly() has been replaced with AssemblyTools.GetApplicationAssembly(); GetExecutableName() and GetExecutablePath() have removed. * All of the constant expressions on the MetaBuilderBase (e.g. EndOfTimeExpression) are obsolete. Instead, use MetaBuilderBase.ExpressionFactory.Constants.EndOfTime instead. * All of the global values on MetaObjectDescriptionExtensions are obsolete; instead, use the IMetaObjectFormatterSettings from the IOC to change settings on startup. * Similarly, the set of extension methods that included GetShortDescription() has been moved to the IMetaObjectFormatter. Obtain an instance from the IOC, as usual. ]]> https://www.earthli.com/news/view_article.php?id=3194 2015-12-28T10:40:24+01:00 https://earthli.com/users/marco

is also available for those with access to the Encodo issue tracker. [Highlights] In the "beta1" and "beta2" release notes, we read about changes to configuration, dependency reduction, the data... ]]>

is also available for those with access to the Encodo issue tracker. [Highlights] In the "beta1" and "beta2" release notes, we read about changes to configuration, dependency reduction, the data driver architecture, DDL commands, security and access control in web applications and a new code-generation format. In 2.0 final -- which was actually released internally on November 13th, 2015 (a Friday) -- we made the following additional improvements: * Moved the metadata table maintained for the schema-migrator to a proper Quino module. ("QNO-4741" ) * Rebuilt the logging and messaging API and drastically simplified the implementation throughout ("QNO-4688" w/sub-tasks, "QNO-4954" ) * Split Encodo and Quino into dozens of new, independent and properly decoupled assemblies ("QNO-4678" , "QNO-4672" , "QNO-4670" , "QNO-4376" , "QNO-4920" , "QNO-4926" ) * Rebuilt the configuration and application-startup API ("QNO-4855" , "QNO-4051" , "QNO-4895" , "QNO-4931" , "QNO-4930" , "QNO-4949" , "QNO-4659" w/sub-tasks, "QNO-4950" , "QNO-4857" , "QNO-4910" , "QNO-4934" , "QNO-4898" , "QNO-4935" , "QNO-4937" ) * Changed delivery and deployment for Quino and all products to Nuget packages ("QNO-4916" ) * Added scripting and support for continuous integration and deployment of Quino packages and symbols via TeamCity ("QNO-4871" , "QNO-4932" , "QNO-3437" , "QNO-4433" , "QNO-4494" , "QNO-4871" ) * Restructured and refactored the standard testing base-classes and testing support for Quino products ("QNO-4963" ) * Improved and fixed code-generation for both v1 and v2 formats ("QNO-4804" , "QNO-4828" , "QNO-4897" ) * Refactored the application server API to improve decoupling and extensibility ("QNO-4927" ) These notes are being published for completeness and documentation. The first publicly available release of Quino 2.x will be 2.1 or higher (release notes coming soon). [Breaking changes] [image]As we've mentioned before, this release is absolutely merciless in regard to backwards compatibility. Old code is not retained as Obsolete. Instead, a project upgrading to 2.0 will encounter compile errors. The following notes serve as an incomplete guide that will help you upgrade a Quino-based product. As I wrote in the release notes for "beta1" and "beta2" , if you arm yourself with a bit of time, ReSharper and the release notes (and possibly keep an Encodo employee on speed-dial), the upgrade is not difficult. It consists mainly of letting ReSharper update namespace references for you. [Global Search/Replace] Instead of going through the errors (example shown to the right) one by one, you can take care of a lot of errors with the following search/replace pairs. * Encodo.Quino.Data.Persistence => Encodo.Quino.Data * IMetaApplication => IApplication * ICoreApplication => IApplication * GetServiceLocator() => GetServices() * MetaMethodTools.GetInstance => DataMetaMethodExtensions.GetInstance * application.ServiceLocator.GetInstance => application.GetInstance * Application.ServiceLocator.GetInstance => Application.GetInstance * application.ServiceLocator => application.GetServices() * Application.ServiceLocator => Application.GetServices() * application.Recorder => application.GetLogger() * Application.Recorder => Application.GetLogger() * session.GetRecorder() => session.GetLogger() * Session.GetRecorder() => Session.GetLogger() * Session.Application.Recorder => Session.GetLogger() * FileTools.Canonicalize() => PathTools.Normalize() * application.Messages => application.GetMessageList() * Application.Messages => Application.GetMessageList() * ServiceLocator.GetInstance => Application.GetInstance * MetaLayoutTools => LayoutConstants * GlobalContext.Instance.Application.Configuration.Model => GlobalContext.Instance.Application.GetModel() * IMessageRecorder => ILogger * GetUseReleaseSettings() => IsInReleaseMode() * ReportToolsDX => ReportDxExtensions Although you can't just search/replace everything, it gets you a long way. [Model-Building Fixes] These replacement pairs, while not recommended for global search/replace, are a handy guide for how the API has generally changed. * *Generator => *Builder * SetUpForModule => CreateModule * Builder.SetElementVisibility(prop, true) => prop.Show() * Builder.SetElementVisibility(prop, false) => prop.Hide() * Builder.SetElementControlIdentifier(prop, ControlIdentifiers => prop.SetInputControl(ControlIdentifiers * Builder.SetPropertyHeightInPixels(prop, 200); => prop.SetHeightInPixels(200); Constructing a module has also changed. Instead of using the following syntax, var module = Builder.SetUpForModule(Name, "ApexClearing.Alps.Core", Name, true); Replace it with the following direct replacement, var module = Builder.CreateModule(Name, "ApexClearing.Alps.Core", Name); Or use this replacement, with the recommended style for the v2 format (no more class prefix for generated classes and a standard namespace): var module = Builder.CreateModule(Name, typeof(AuditModuleBuilder).GetParentNamespace()); [Standard Modules (e.g. Reporting, Security, etc.)] Because of how the module class-names have changed, the standard module ORM classes all have different names. The formula is that the ORM class-name is no longer prepended its module name. * ReportsReportDefinition => ReportDefinition * SecurityUser => User * And so on... Furthermore, all modules have been converted to use the v2 code-generation format, which has the metadata separate from the ORM object. Therefore, instead of referencing metadata using the ORM class-name as the base, you use the module name as the base. * ReportReportDefinition.Fields.Name => ReportModule.ReportDefinition.Name.Identifier * ReportReportDefinition.MetaProperties.Name => ReportModule.ReportDefinition.Name * ReportReportDefinition.Metadata => ReportModule.ReportDefinition.Metadata * And so on... There's an upcoming article that will show more examples of the improved flexibility and capabilities that come with the v2-metadata. [Action names] The standard action names have moved as well. * ActionNames => ApplicationActionNames * MetaActionNames => MetaApplicationActionNames Any other, more rarely used action names have been moved back to the actions themselves, so for example SaveApplicationSettingsAction.ActionName If you created any actions of your own, then the API there has changed as well. As previously documented in "API Design: To Generic or not Generic? (Part II)" , instead of overriding the following method, protected override int DoExecute(IApplication application, ConfigurationOptions options, int currentResult) { base.DoExecute(application, options, currentResult); } you instead override in the following way, public override void Execute() { base.Execute(); } [Using NuGet] If you're already using Visual Studio 2015, then the NuGet UI is a good choice for managing packages. If you're still on Visual Studio 2013, then the UI there is pretty flaky and we recommend using the console. The examples below assume that you have configured a source called "Local Quino" (e.g. a local folder that holds the nupkg files for Quino). install-package Quino.Data.PostgreSql.Testing -ProjectName Punchclock.Core.Tests -Source "Local Quino" install-package Quino.Server -ProjectName Punchclock.Server -Source "Local Quino" install-package Quino.Console -ProjectName Punchclock.Server -Source "Local Quino" install-package Quino.Web -ProjectName Punchclock.Web.API -Source "Local Quino" [Debugging Support] We recommend using Visual Studio 2015 if at all possible. Visual Studio 2013 is also supported, but we have all migrated to 2015 and our knowhow about 2013 and its debugging idiosyncrasies will deteriorate with time. These are just brief points of interest to get you set up. As with the NuGet support, these instructions are subject to change as we gain more experience with debugging with packages as well. * Hook up to a working symbol-source server (e.g. TeamCity) * Get the local sources for your version * If you don't have a source server or it's flaky, then get the PDBs for the Quino version you're using (provided in Quino.zip as part of the package release) * Add the path to the PDBs to your list of symbol sources in the VS debugging options * Tell Visual Studio where the sources are when it asks during debugging * Tell R# how to map from the source folder (c:\BuildAgent\work\9a1bb0adebb73b1f for Quino 2.0.0-1765) to the location of your sources Quino packages are no different than any other NuGet packages. We provide both standard packages as well as packages with symbols and sources. Any complications you encounter with them are due to the whole NuGet experience still being a bit in-flux in the .NET world. An upcoming post will provide more detail and examples. [Creating Nuget Packages] We generally use our continuous integration server to create packages, but you can also create packages locally (it's up to you to make sure the version number makes sense, so be careful). These instructions are approximate and are subject to change. I provide them here to give you an idea of how packages are created. If they don't work, please contact Encodo for help. * Open PowerShell * Change to the %QUINO_ROOT%\src directory * Run nant build pack to build Quino and packages * Set up a local NuGet Source name "Local Quino" to %QUINO_ROOT%\nuget (one-time only) * Change to the directory where your Quino packages are installed for your solution. * Delete all of the Encodo/Quino packages * Execute nant nuget from your project directory to get the latest Quino build from your local folder ]]> https://www.earthli.com/news/view_article.php?id=3189 2015-12-06T11:57:57+01:00 https://earthli.com/users/marco

I created for the impending release of Quino 2. [Environment] * Windows 8.1 Enterprise * Visual Studio 2015 * ReSharper 9.2 [Expected behavior] Non-leaf-node base classes should never appear as nodes in test runners. A user should be able to run tests in descendants directly from a fixture or test in the base class. [Observed behavior] Non-leaf-node base classes are shown in the R# test runner in both versions 9 and 10. A user must navigate to the descendant to run a test. The user can no longer run all descendants or a single descendant directly from the test. [Analysis] Relatively recently, in order to better test a misbehaving test runner and accurately report issues to JetBrains, I standardized all tests to the same pattern: * Do not use abstract anywhere (the base classes don't technically need it) * Use the TestFixture attribute only on leaf nodes This worked just fine with ReSharper 8.x but causes strange behavior in both R# 9.x and 10.x. We discovered recently that not only did the test runner act strangely (something that they might fix), but also that the unit-testing integration in the files themselves behaved differently when the base class is abstract (something JetBrains is unlikely to fix). You can see that R# treats a non-abstract class with tests as a testable entity, even when it doesn't actually have a TestFixture attribute and even expects a generic type parameter in order to instantiate. Here it's not working well in either the source file or the test runner. In the source file, you can see that it offers to run tests in a category, but not the tests from actual descendants. If you try to run or debug anything from this menu, it shows the fixture with a question-mark icon and marks any tests it manages to display as inconclusive. This is not surprising, since the test fixture may not be abstract, but does require a type parameter in order to be instantiated. [image] Here it looks and acts correctly: [image] I've reported this issue to JetBrains, but our testing structure either isn't very common or it hasn't made it to their core test cases, because neither 9 nor 10 handles them as well as the 8.x runner did. Now that we're also using TeamCity a lot more to not only execute tests but also to collect coverage results, we'll capitulate and just change our patterns to whatever makes R#/TeamCity the happiest. [Solution] * Make all testing base classes that include at least one {{Test}} or {{Category}} attribute {{abstract}}. Base classes that do not have any testing attributes do not need to be made abstract. Once more to recap our ground rules for making tests: * Include TestFixture only on leafs (classes with no descendants) * You can put Category or Test attributes anywhere in the hierarchy, but need to declare the class as abstract. * Base classes that have no testing attributes do not need to be abstract * If you feel you need to execute tests in both a base class and one of its descendants, then you're probably doing something wrong. Make two descendants of the base class instead. When you make the change, you can see the improvement immediately. [image] -------------------------------------------------------------------------------- [1] ReSharper 10.0 also offers continuous integration, but our experiments with the EAP builds and the first RTM build left us underwhelmed and we downgraded to 9.2 until JetBrains manages to release a stable 10.x. ]]> https://www.earthli.com/news/view_article.php?id=3175 2015-11-28T13:58:45+01:00 https://earthli.com/users/marco

and "API Design: To Generic or not Generic? (Part II)" as well as the three-part series that starts with "Encodo’s configuration library for Quino: part I" . [Quino Execution Stages] The life-cycle of a Quino 2.0 application breaks down into roughly the following stages: 1. Build Application: Register services with the IOC, add objects needed during configuration and add actions to the startup and shutdown lists 2. Load User Configuration: Use non-IOC objects to bootstrap configuration from the command line and configuration files; IOC is initialized and can no longer be modified after action ServicesInitialized 3. Apply Application Configuration: Apply code-based configuration to IOC objects; ends with the ServicesConfigured action 4. Execute: execute the loop, event-handler, etc. 5. Shut Down: dispose of the application, shutting down services in the IOC, setting the exit code, etc. [Stage 1] The first stage is all about putting the application together with calls to Use various services and features. This stage is covered in detail in three parts, starting with "Encodo’s configuration library for Quino: part I" . [Stage 2] Let's tackle this one last because it requires a bit more explanation. [Stage 3] Technically, an application can add code to this stage by adding an IApplicationAction before the ServicesConfigured action. Use the Configure() extension method in stage 1 to configure individual services, as shown below. application.Configure( s => s.Behavior = FileLogBehavior.MultipleFiles ); [Stage 4] The execution stage is application-specific. This stage can be short or long, depending on what your application does. For desktop applications or single-user utilities, stage 4 is executed in application code, as shown below, in the Run method, which called by the ApplicationManager after the application has started. var transcript = new ApplicationManager().Run(CreateApplication, Run); IApplication CreateApplication() { ... } void Run(IApplication application) { ... } If your application is a service, like a daemon or a web server or whatever, then you'll want to execute stages 1--3 and then let the framework send requests to your application's running services. When the framework sends the termination signal, execute stage 5 by disposing of the application. Instead of calling Run, you'll call CreateAndStartupUp. var application = new ApplicationManager().CreateAndStartUp(CreateApplication); IApplication CreateApplication() { ... } [Stage 5] Every application has certain tasks to execute during shutdown. For example, an application will want to close down any open connections to external resources, close file (especially log files) and perhaps inform the user of shutdown. Instead of exposing a specific "shutdown" method, a Quino 2.0 application can simply be disposed to shut it down. If you use ApplicationManager.Run() as shown above, then you're already sorted -- the application will be disposed and the user will be informed in case of catastrophic failure; otherwise, you can shut down and get the final application transcript from the disposed object. application.Dispose(); var transcript = application.GetTranscript(); // Do something with the transcript... [Stage 2 Redux] We're finally ready to discuss stage 2 in detail. An IOC has two phases: in the first phase, the application registers services with the IOC; in the second phase, the application uses services from the IOC. An application should use the IOC as much as possible, so Quino keeps stage 2 as short as possible. Because it can't use the IOC during the registration phase, code that runs in this stage shares objects via a poor-man's IOC built into the IApplication that allows modification and only supports singletons. Luckily, very little end-developer application code will ever need to run in this stage. It's nevertheless interesting to know how it works. Obviously, any code in this stage that uses the IOC will cause it to switch from phase one to phase two and subsequent attempts to register services will fail. Therefore, while application code in stage 2 has to be careful, you don't have to worry about not knowing you've screwed up. Why would we have this stage? Some advocates of using an IOC claim that everything should be configured in code. However, it's not uncommon for applications to want to run very differently based on command-line or other configuration parameters. The Quino startup handles this by placing the following actions in stage 2: * Parse and apply command-line * Import and apply external configuration (e.g. from file) An application is free to insert more actions before the ServicesInitialized action, but they have to play by the rules outlined above. ["Single" objects] Code in stage 2 shares objects by calling SetSingle() and GetSingle(). There are only a few objects that fall into this category. The calls UseCore() and UseApplication() register most of the standard objects used in stage 2. Actually, while they're mostly used during stage 2, some of them are also added to the poor man's IOC in case of catastrophic failure, in which case the IOC cannot be assumed to be available. A good example is the IApplicationCrashReporter. [Executing Stages] Before listing all of the objects, let's take a rough look at how a standard application is started. The following steps outline what we consider to be a good minimum level of support for any application. Of course, the Quino configuration is modular, so you can take as much or as little as you like, but while you can use a naked Application -- which has absolutely nothing registered -- and you can call UseCore() to have a bit more -- it registers a handful of low-level services but no actions -- we recommend calling at least UseApplication() to adds most of the functionality outlined below. 1. Create application: This involves creating the IOC and most of the IOC registration as well as adding most of the application startup actions (stage 1) 2. Set debug mode: Get the final value of RunMode from the IRunSettings to determine if the application should catch all exceptions or let them go to the debugger. This involves getting the IRunSettings from the application and getting the final value using the IApplicationManagerPreRunFinalizer. This is commonly an implementation that can allows setting the value of RunMode from the command-line in debug builds. This further depends on the ICommandSetManager (which depends on the IValueTools) and possibly the ICommandLineSettings (to set the CommandLineConfigurationFilename if it was set by the user). 3. Process command line: Set the ICommandProcessingResult, possibly setting other values and adding other configuration steps to the list of startup actions (e.g. many command-line options are switches that are handled by calling Configure() where TSettings is the configuration object in the IOC to modify). 4. Read configuration file: Load the configuration data into the IConfigurationDataSettings, involving the ILocationManager to find configuration files and the ITextValueNodeReader to read them. 5. The ILogger is used throughout by various actions to log application behavior 6. If there is an unhandled error, the IApplicationCrashReporter uses the IFeedback or the ILogger to notify the user and log the error 7. The IInMemoryLogger is used to include all in-memory messages in the IApplicationTranscript The next section provides detail to each of the individual objects referenced in the workflow above. [Available Objects] You can get any one of these objects from the IApplication in at least two ways, either by using GetSingle() (safe in all situations) or GetInstance() (safe only in stage 3 or later) or there's almost always a method which starts with "Use" and ends in the service name. The example below shows how to get the ICommandSetManager [2] if you need it. application.GetCommandSetManager(); application.GetSingle(); // Prefer the one above application.GetInstance(); All three calls return the exact same object, though. The first two from the poor-man's IOC; the last from the real IOC. Only applications that need access to low-level objects or need to mess around in stage 2 need to know which objects are available where and when. Most applications don't care and will just always use GetInstance(). The objects in the poor-man's IOC are listed below. [Core] * IValueTools: converts values; used by the command-line parser, mostly to translate enumerate values and flags * ILocationManager: an object that manages aliases for file-system locations, like "Configuration", from which configuration files should be loaded or "UserConfiguration" where user-specific overlay configuration files are stored; used by the configuration loader * ILogger: a reference to the main logger for the application * IInMemoryLogger: a reference to an in-memory message store for the logger (used by the ApplicationManager to retrieve the message log from a crashed application) * IMessageFormatter: a reference to the object that formats messages for the logger [Command line] * ICommandSetManager: sets the schema for a command line; used by the command-line parser * ICommandProcessingResult: contains the result of having processed the command line * ICommandLineSettings: defines the properties needed to process the command line (e.g. the Arguments and CommandLineConfigurationFilename, which indicates the optional filename to use for configuration in addition to the standard ones) [Configuration] * IConfigurationDataSettings: defines the ConfigurationData which is the hierarchical representation of all configuration data for the application as well as the MainConfigurationFilename from which this data is read; used by the configuration-loader * ITextValueNodeReader: the object that knows how to read ConfigurationData from the file formats supported by the application [3]; used by the configuration-loader [Run] * IRunSettings: an object that manages the RunMode ("release" or "debug"), which can be set from the command line and is used by the ApplicationManager to determine whether to use global exception-handling * IApplicationManagerPreRunFinalizer: a reference to an object that applies any options from the command line before the decision of whether to execute in release or debug mode is taken. * IApplicationCrashReporter: used by the ApplicationManager in the code surrounding the entire application execution and therefore not guaranteed to have a usable IOC available * IApplicationDescription: used together with the ILocationManager to set application-specific aliases to user-configuration folders (e.g. AppData\{CompanyTitle}\{ApplicationTitle}) * IApplicationTranscript: an object that records the last result of having run the application; returned by the ApplicationManager after Run() has completed, but also available through the application object returned by CreateAndStartUp() to indicate the state of the application after startup. Each of these objects has a very compact interface and has a single responsibility. An application can easily replace any of these objects by calling UseSingle() during stage 1 or 2. This call sets the object in both the poor-man's IOC as well as the real one. For those rare cases where a non-IOC singleton needs to be set after the IOC has been finalized, the application can call SetSingle(), which does not touch the IOC. This feature is currently used only to set the IApplicationTranscript, which needs to happen even after the IOC registration is complete. -------------------------------------------------------------------------------- [1] Two large customer solutions, two medium-sized internal solutions (Punchclock and JobVortex) as well as the Demo/Sandbox solution. These solutions include the gamut of application types: * 3 ASP.NET MVC applications * 2 ASP.NET WebAPI applications * 2 Windows services * 3 Winform/DevExpress applications * 2 Winform/DevExpress utilities * 4 Console applications and utilities [1] I originally used ITextValueNodeReader as an example, but that's one case where the recommended call doesn't match 1-to-1 with the interface name. application.GetSingle(); application.GetInstance(); application.GetConfigurationDataReader(); // Recommended [1] Currently only XML, but "JSON" is on the way when someone gets a free afternoon. ]]> https://www.earthli.com/news/view_article.php?id=3173 2015-11-23T22:31:29+01:00 https://earthli.com/users/marco

: [image] This screenshot was captured after a few unneeded interfaces had already been removed. As you can see by the class names, we'd struggled heroically to deal with the complexity that arises when you use inheritance rather than composition. The state-handling was welded onto an authentication-enabled server, and the base machinery for supporting authentication was spread across three hierarchy layers. The hierarchy only hints at composition in its naming: the "Stateful" part of the class name CoreStatefulHttpServerBase had already been moved to a state provider and a state creator in previous versions. That support is unchanged in the 2.0 version. [Implementation Layers] We mentioned above that implementation was "spread across three hierarchy layers". There's nothing wrong with that, in principle. In fact, it's a good idea to encapsulate higher-level patterns in a layer that doesn't introduce too many dependencies and to introduce dependencies in other layers. This allows applications not only to be able to use a common implementation without pulling in unwanted dependencies, but also to profit from the common tests that ensure the components works as advertised. In Quino, the following three layers are present in many components: 1. Abstract: a basic encapsulation of a pattern with almost no dependencies (generally just Encodo.Core). 2. Standard: a functional implementation of the abstract pattern with dependencies on non-metadata assemblies (e.g. Encodo.Application, Encodo.Connections and so on) 3. Quino: an enhancement of the standard implementation that makes use of metadata to fill in implementation left abstract in the previous layer. Dependencies can include any of the Quino framework assemblies (e.g. Quino.Meta, Quino.Application and so on). [The New Hotness [1]] The diagram below shows the new hotness in Quino 2. [2] [image] The hierarchy is now extremely flat. There is an IServer interface and a Server implementation, both generic in TListener, of type IServerListener. The server manages a single instance of an IServerListener. The listener, in turn, has an IHttpServerRequestHandler, the main implementation of which uses an IHttpServerAuthenticator. As mentioned above, the IServerStateProvider is included in this diagram, but is unchanged from Quino 2.0-beta3, except that it is now used by the request handler rather than directly by the server. You can see how the abstract layer is enhanced by an HTTP-specific layer (the Encodo.Server.Http namespace) and the metadata-specific layer is nice encapsulated in three classes in the Quino.Server assembly. [Server Components and Flow] This type hierarchy has decoupled the main elements of the workflow of handling requests for a server: * The server manages listeners (currently a single listener), created by a listener factory * The listener, in turn, dispatches requests to the request handler * The request handler uses the route handler to figure out where to direct the request * The route handler uses a registry to map requests to response items * The request handler asks the state provider for the state for the given request * The state provider checks its cache for the state (the default support uses persistent states to cache sessions for a limited time); if not found, it creates a new one * Finally, the request handler checks whether the user for the request is authenticated and/or authorized to execute the action and, if so, executes the response items It is important to note that this behavior is unchanged from the previous version -- it's just that now each step is encapsulated in its own component. The components are small and easily replaced, with clear and concise interfaces. Note also that the current implementation of the request handler is for HTTP servers only. Should the need arise, however, it would be relatively easy to abstract away the HttpListener dependency and generalize most of the logic in the request handler for any kind of server, regardless of protocol and networking implementation. Only the request handler is affected by the HTTP dependency, though: authentication, state-provision and listener-management can all be re-used as-is. Also of note is that the only full-fledged implementation is for metadata-based applications. At the bottom of the diagram, you can see the metadata-specific implementations for the route registry, state provider and authenticator. This is reflected in the standard registration in the IOC. These are the service registrations from Encodo.Server: return handler .RegisterSingle() .RegisterSingle, HttpServerListenerFactory>() .Register>(); And these are the service registrations from Quino.Server: handler .RegisterSingle, StandardMetaServerRouteRegistry>() .RegisterSingle, MetaPersistentServerStateProvider>() .RegisterSingle, MetaServerStateCreator>() .RegisterSingle, MetaHttpServerAuthenticator>() .RegisterSingle>() As you can see, the registration is extremely fine-grained and allows very precise customization as well as easy mocking and testing. -------------------------------------------------------------------------------- [1] Any Men in Black fans out there? Tommy Lee Jones was "old and busted" while Will Smith was "the new hotness"? No? Just me? All righty then... [1] This diagram brought to you by the diagramming and architecture tools in ReSharper 9.2. Just select the files or assemblies you want to diagram in the Solution Explorer and choose the option to show them in a diagram. You can right-click any type or assembly to show dependent or referenced modules or types. For type diagrams , you can easily control which relationships are to be shown (e.g. I hide aggregations to avoid clutter) and how the elements are to be grouped (e.g. I grouped by namespace to include the boxes in my diagram). ]]> https://www.earthli.com/news/view_article.php?id=3169 2015-10-16T11:44:35+02:00 https://earthli.com/users/marco

In this article, I'll discuss both the high-level and low-level workflows I used with NDepend to efficiently clear up these cycles. Please note that what follows is a description of how I have used... ]]>

In this article, I'll discuss both the high-level and low-level workflows I used with NDepend to efficiently clear up these cycles. Please note that what follows is a description of how I have used the tool -- so far -- to get my very specific tasks accomplished. If you're looking to solve other problems or want to solve the same problems more efficiently, you should take a look at the "official NDepend documentation" . [What were we doing?] To recap briefly: we are reducing dependencies among top-level namespaces in two large assemblies, in order to be able to split them up into multiple assemblies. The resulting assemblies will have dependencies on each other, but the idea is to make at least some parts of the Encodo/Quino libraries opt-in. [The plan of attack] On a high-level, I tackled the task in the following loosely defined phases. Remove direct, root-level dependencies This is the big first step -- to get rid of the little black boxes. I made NDepend show only direct dependencies at first, to reduce clutter. More on specific techniques below. Remove indirect dependencies [image]Crank up the magnification to show indirect dependencies as well. This will will help you root out the remaining cycles, which can be trickier if you're not showing enough detail. On the contrary, if you turn on indirect dependencies too soon, you'll be overwhelmed by darkness (see the depressing initial state of the Encodo assembly to the right). Examine dependencies between root-level namespaces Even once you've gotten rid of all cycles, you may still have unwanted dependencies that hinder splitting namespaces into the desired constellation of assemblies. For example, the plan is to split all logging and message-recording into an assembly called Encodo.Logging. However, the IRecorder interface (with a single method, Log()) is used practically everywhere. It quickly becomes necessary to split interfaces and implementation -- with many more potential dependencies -- into two assemblies for some very central interfaces and support classes. In this specific case, I moved IRecorder to Encodo.Core. Even after you've conquered the black hole, you might still have quite a bit of work to do. Never fear, though: NDepend is there to help root out those dependencies as well. Examine cycles in non-root namespaces Because we can split off smaller assemblies regardless, these dependencies are less important to clean up for our current purposes. However, once this code is packed into its own assembly, its namespaces become root namespaces of their own and -- voila! you have more potentially nasty dependencies to deal with. Granted, the problem is less severe because you're dealing with a logically smaller component. In Quino, use non-root namespaces more for organization and less for defining components. Still, cycles are cycles and they're worth examining and at least plucking the low-hanging fruit. [Removing root-level namespace cycles] With the high-level plan described above in hand, I repeated the following steps for the many dependencies I had to untangle. Don't despair if it looks like your library has a ton of unwanted dependencies. If you're smart about the ones you untangle first, you can make excellent -- and, most importantly, rewarding -- progress relatively quickly. [1] 1. Show the dependency matrix 2. Choose the same assembly in the row and column 3. Choose a square that's black 4. Click the name of the namespace in the column to show sub-namespaces 5. Do the same in a row 6. Keep zooming until you can see where there are dependencies that you don't want 7. Refactor/compile/run NDepend analysis to show changes 8. GOTO 1 [Once again, with pictures!] The high-level plan of attack sounded interesting, but might have left you cold with its abstraction. Then there was the promise of detail with a focus on root-level namespaces, but alas, you might still be left wondering just how exactly do you reduce these much-hated cycles? I took some screenshots as I worked on Quino, to document my process and point out parts of NDepend I thought were eminently helpful. [Show only namespaces] [image][image]I mentioned above that you should "[k]eep zooming in", but how do you do that? A good first step is to zoom all the way out and show only direct namespace dependencies. This focuses only on using references instead of the much-more frequent member accesses. In addition, I changed the default setting to show dependencies in only one direction -- when a column references a row (blue), but not vice versa (green). As you can see, the diagrams are considerably less busy than the one shown above. Here, we can see a few black spots that indicate cycles, but it's not so many as to be overwhelming. [2] You can hover over the offending squares to show more detail in a popup. [Show members] [image][image]If you don't see any more cycles between namespaces, switch the detail level to "Members". Another very useful feature is to "Bind Matrix", which forces the columns and rows to be shown in the same order and concentrates the cycles in a smaller area of the matrix. As you can see in the diagram, NDepend then highlights the offending area and you can even click the upper-left corner to focus the matrix only on that particular cycle. [Drill down to classes] [image][image]Once you're looking at members, it isn't enough to know just the namespaces involved -- you need to know which types are referencing which types. The powerful matrix view lets you drill down through namespaces to show classes as well. If your classes are large -- another no-no, but one thing at a time -- then you can drill down to show which method is calling which method to create the cycle. In the screenshot to the right, you can see where I had to do just that in order to finally figure out what was going on. In that screenshot, you can also see something that I only discovered after using the tool for a while: the direction of usage is indicated with an arrow. You can turn off the tooltips -- which are informative, but can be distracting for this task -- and you don't have to remember which color (blue or green) corresponds to which direction of usage. [Indirect dependencies] [image][image]Once you've drilled your way down from namespaces-only to showing member dependencies, to focusing on classes, and even members, your diagram should be shaping up quite well. On the right, you'll see a diagram of all direct dependencies for the remaining area with a problem. You don't see any black boxes, which means that all direct dependencies are gone. So we have to turn up the power of our microscope further to show indirect dependencies. On the left, you can see that the scary, scary black hole from the start of our journey has been whittled down to a small, black spot. And that's with all direct and indirect dependencies as well as both directions of usage turned on (i.e. the green boxes are back). This picture is much more pleasing, no? [Queries and graphs] [image][image][image]For the last cluster of indirect dependencies shown above, I had to unpack another feature: NDepend queries: you can select any element and run a query to show using/used by assemblies/namespaces. [3] The results are shown in a panel, where you can edit the query to see live updates immediately. Even with a highly zoomed-in view on the cycle, I still couldn't see the problem, so I took NDepend's suggestion and generated a graph of the final indirect dependency between Culture and Enums (through Expression). At this zoom level, the graph becomes more useful (for me) and illuminates problems that remain muddy in the matrix (see right). [Crossing the finish line] In order to finish the job efficiently, here are a handful of miscellaneous tips that are useful, but didn't fit into the guide above. [image] * I set NDepend to automatically re-run an analysis on a successful build. The matrix updates automatically to reflect changes from the last analysis and won't lose your place. * If you have ReSharper, you'll generally be able to tell whether you've fixed the dependencies because the usings will be grayed out in the offending file. You can make several fixes at once before rebuilding and rerunning the analysis. * At higher zoom levels (e.g. having drilled down to methods), it is useful to toggle display of row dependencies back on because the dependency issue is only clear when you see the one green box in a sea of blue. * Though Matrix Binding is useful for localizing, remember to toggle it off when you want to drill down in the row independently of the namespace selected in the column. And BOOM! just like that [4], phase 1 (root namespaces) for Encodo was complete! Now, on to Quino.dll... [Conclusion] [image]Depending on what shape your library is in, do not underestimate the work involved. Even with NDepend riding shotgun and barking out the course like a rally navigator, you still have to actually make the changes. That means lots of refactoring, lots of building, lots of analysis, lots of running tests and lots of reviews of at-times quite-sweeping changes to your code base. The destination is worth the journey, but do not embark on it lightly -- and don't forget to bring the right tools. [5] -------------------------------------------------------------------------------- [1] This can be a bit distracting: you might get struck trying to figure out which of all these offenders to fix first. [1] I'm also happy to report that my initial forays into maintaining a relatively clean library -- as opposed to cleaning it -- with NDepend have been quite efficient. [1] And much more: I don't think I've even scratched the surface of the analysis and reporting capabilities offered by this ability to directly query the dependency data. [1] I'm just kidding. It was a lot of time-consuming work. [1] In this case, in case it's not clear: NDepend for analysis and good ol' ReSharper for refactoring. And ReSharper's new(ish) architecture view is also quite good, though not even close to detailed enough to replace NDepend: it shows assembly-level dependencies only. ]]> https://www.earthli.com/news/view_article.php?id=3167 2015-10-04T07:44:39+02:00 https://earthli.com/users/marco

: I really, really like the depth of insight NDepend gives me into my code. I find myself thinking "SOLID" much more often when I have NDepend shaking its head sadly at me, tsk-tsking at all of the dependency snarls I've managed to build. * It's fast and super-reliable. I can work these checks into my workflow relatively easily. * I'm using the matrix view a lot more than the graphs because even NDepend recommends I don't use a graph for the number of namespaces/classes I'm usually looking at * Where the graph view is super-useful is for examining *indirect* dependencies, which are harder to decipher with the graph * I've found so many silly mistakes/lazy decisions that would lead to confusion for developers new to my framework * I'm spending so much time with it and documenting my experiences because I want more people at my company to use it * I haven't even scratched the surface of the warnings/errors but want to get to that, as well (the Dashboard tells me of 71 rules violated; 9 critical; I'm afraid to look :-) [Use Cases] Before I get more in-depth with NDepend, please note that there at least two main use cases for this tool [4]: 1. Clean up a project or solution that has never had a professional dependency checkup 2. Analyze and maintain separation and architectural layers in a project or solution These two use cases are vastly different. The first is like cleaning a gas-station bathroom for the first time in years; the second is more like the weekly once-over you give your bathroom at home. The tools you'll need for the two jobs are similar, but quite different in scope and power. The same goes for NDepend: how you'll use it to claw your way back to architectural purity is different than how you'll use it to occasionally clean up an already mostly-clean project. Quino is much better than it was the last time we peeked under the covers with NDepend, but we're still going to need a bucket of industrial cleaner before we're done. [5] The first step is to make sure that you're analyzing the correct assemblies. Show the project properties to see which assemblies are included. You should remove all assemblies from consideration that don't currently interest you (especially if your library is not quite up to snuff, dependency-wise; afterwards, you can leave as many clean assemblies in the list as you like). [6] [Industrial-strength cleaner for Quino] Running an analysis with NDepend 6 generates a nice report, which includes the following initial dependency graph for the assemblies. [image] As you can see, Encodo and Quino depend only on system assemblies, but there are components that pull in other references where they might not be needed. The initial dependency matrices for Encodo and Quino both look much better than they did when I last generated one. The images below show what we have to work with in the Encodo and Quino assemblies. [image][image] It's not as terrible as I've made out, right? There is far less namespace-nesting, so it's much easier to see where the bidirectional dependencies are. There are only a handful of cyclic dependencies in each library, with Encodo edging out Quino because of (A) the nature of the code and (B) I'd put more effort into Encodo so far. I'm not particularly surprised to see that this is relatively clean because we've put effort into keeping the external dependencies low. It's the internal dependencies in Encodo and Quino that we want to reduce. [Small and Focused Assemblies] [image][image][image]The goal, as stated in the title of this article, is to split Encodo and Quino into separate assemblies. While removing cyclic dependencies is required for such an operation, it's not sufficient. Even without cycles, it's still possible that a given assembly is still too dependent on other assemblies. Before going any farther, I'm going to list the assemblies we'd like to have. By "like to have", I mean the list that we'd originally planned plus a few more that we added while doing the actual splitting. [7] The images on the right show the assemblies in Encodo, Quino and a partial overview of the dependency graph (calculated with the ReSharper Architecture overview rather than with NDepend, just for variety). Of these, the following assemblies and their dependencies are of particular interest [8]: * Encodo.Core: System dependencies only * Encodo.Application: basic application support [9] * Encodo.Application.Standard: configuration methods for non-metadata applications that don't want to pick and choose packages/assemblies * Encodo.Expressions: depends only on Encodo.Core * Quino.Meta: depends only on Encodo.Core and Encodo.Expressions * Quino.Meta.Standard: Optional, but useful metadata extensions * Quino.Application: depends only on Encodo.Application and Quino.Meta * Quino.Application.Standard: configuration methods for metadata applications that don't want to pick and choose packages/assemblies * Quino.Data: depends on Quino.Application and some Encodo.* assemblies * Quino.Schema: depends on Quino.Data This seems like a good spot to stop, before getting into the nitty-gritty detail of how we used NDepend in practice. In the next article, I'll discuss both the high-level and low-level workflows I used with NDepend to efficiently clear up these cycles. Stay tuned! -------------------------------------------------------------------------------- [1] Release notes for 2.0 betas: * "v2.0-beta1: Configuration, services and web" * "v2.0-beta2: Code generation, IOC and configuration" Articles about design: * "Encodo’s configuration library for Quino: part I" * "Encodo’s configuration library for Quino: part II" * "Encodo’s configuration library for Quino: part III" * "API Design: Running an Application (Part I)" * "API Design: To Generic or not Generic? (Part II)" [1] I published a two-parter in August and November of 2014. * "The Road to Quino 2.0: Maintaining architecture with NDepend (part I)" * "The Road to Quino 2.0: Maintaining architecture with NDepend (part II)" [1] You can see a lot of the issues associated with these changes in the release notes for "Quino 2.0-beta1" (mostly the first point in the "Highlights" section) and "Quino 2.0-beta2" (pretty much all of the points in the "Highlights" section). [1] I'm sure there are more, but those are the ones I can think of that would apply to my project (for now). [1] ...to stretch the gas-station metaphor even further. [1] Here I'm going to give you a tip that confused me for a while, but that I think was due to particularly bad luck and is actually quite a rare occurrence. If you already see the correct assemblies in the list, you should still check that NDepend picked up the right paths. That is, if you haven't followed the advice in NDepend's white paper and still have a different bin folder for each assembly, you may see something like the following in the tooltip when you hover over the assembly name: "Several valid .NET assemblies with the name {Encodo} have been found. They all have the same version. the one with the biggest file has been chosen." If NDepend has accidentally found an older copy of your assembly, you must delete that assembly. Even if you add an assembly directly, NDepend will not honor the path from which you added it. This isn't as bad as it sounds, since it's a very strange constellation of circumstances that led to this assembly hanging around anyway: * The project is no longer included in the latest Quino but lingers in my workspace * The version number is unfortunately the same, even though the assembly is wildly out of date I only noticed because I knew I didn't have that many dependency cycles left in the Encodo assembly. [1] Especially for larger libraries like Quino, you'll find that your expectations about dependencies between modules will be largely correct, but will still have gossamer filaments connecting them that prevent a clean split. In those cases, we just created new assemblies to hold these common dependencies. Once an initial split is complete, we'll iterate and refactor to reduce some of these ad-hoc assemblies.[1] Screenshots, names and dependencies are based on a pre-release version of Quino, so while the likelihood is small, everything is subject to change. [1] Stay tuned for an upcoming post on the details of starting up an application, which is the support provided in Encodo.Application. ]]> https://www.earthli.com/news/view_article.php?id=3166 2015-09-26T11:27:08+02:00 https://earthli.com/users/marco

, where we laid some groundwork about the state machine that is the startup/execution/shutdown feature of Quino. As we discussed, this part of the API still suffers from "several places where generic TApplication parameters"... ]]>

, where we laid some groundwork about the state machine that is the startup/execution/shutdown feature of Quino. As we discussed, this part of the API still suffers from "several places where generic TApplication parameters [are] cluttering the API". In this article, we'll take a closer look at different design approaches to this concrete example -- and see how we decided whether to use generic type parameters. [Consistency through Patterns and API] Any decision you take with a non-trivial API is going to involve several stakeholders and aspects. It's often not easy to decide which path is best for your stakeholders and your product. "For any API you design, consider how others are likely to extend it -- and whether your pattern is likely to deteriorate from neglect." For any API you design, consider how others are likely to extend it -- and whether your pattern is likely to deteriorate from neglect. Even a very clever solution has to be balanced with simplicity and elegance if it is to have a hope in hell of being used and standing the test of time. In Quino 2.0, the focus has been on ruthlessly eradicating properties on the IApplication interface as well as getting rid of the descendant interfaces, ICoreApplication and IMetaApplication. Because Quino now uses a pattern of placing sub-objects in the IOC associated with an IApplication, there is far less need for a generic TApplication parameter in the rest of the framework. See "Encodo’s configuration library for Quino: part I" for more information and examples. This focus raised an API-design question: if we no longer want descendant interfaces, should we eliminate parameters generic in that interface? Or should we continue to support generic parameters for applications so that the caller will always get back the type of application that was passed in? Before getting too far into the weeds [1], let's look at a few concrete examples to illustrate the issue. [Do Fluent APIs require generic return-parameters?] As discussed in "Encodo’s configuration library for Quino: part III" in detail, Quino applications are configured with the "Use*" pattern, where the caller includes functionality in an application by calling methods like UseRemoteServer() or UseCommandLine(). The latest version of this API pattern in Quino recommends returning the application that was passed in to allow chaining and fluent configuration. For example, the following code chains the aforementioned methods together without creating a local variable or other clutter. return new CodeGeneratorApplication().UseRemoteServer().UseCommandLine(); What should the return type of such standard configuration operations be? Taking a method above as an example, it could be defined as follows: public static IApplication UseCommandLine(this IApplication application, string[] args) { ... } This seems like it would work fine, but the original type of the application that was passed in is lost, which is not exactly in keeping with the fluent style. In order to maintain the type, we could define the method as follows: public static TApplication UseCommandLine(this TApplication application, string[] args) where TApplication : IApplication { ... } This style is not as succinct but has the advantage that the caller loses no type information. On the other hand, it's more work to define methods in this way and there is a strong likelihood that many such methods will simply be written in the style in the first example. "Generics definitely offer advantages, but it remains to be seen how much those advantages are worth." Why would other coders do that? Because it's easier to write code without generics, and because the stronger result type is not needed in 99% of the cases. If every configuration method expects and returns an IApplication, then the stronger type will never come into play. If the compiler isn't going to complain, you can expect a higher rate of entropy in your API right out of the gate. One way the more-derived type would come in handy is if the caller wanted to define the application-creation method with their own type as a result, as shown below: private static CodeGeneratorApplication CreateApplication() { return new CodeGeneratorApplication().UseRemoteServer().UseCommandLine(); } If the library methods expect and return IApplication values, the result of UseCommandLine() will be IApplication and requires a cast to be used as defined above. If the library methods are defined generic in TApplication, then everything works as written above. This is definitely an advantage, in that the user gets the exact type back that they created. Generics definitely offer advantages, but it remains to be seen how much those advantages are worth. [2] [Another example: The IApplicationManager] Before we examine the pros and cons further, let's look at another example. In Quino 1.x, applications were created directly by the client program and passed into the framework. In Quino 2.x, the IApplicationManager is responsible for creating and executing applications. A caller passes in two functions: one to create an application and another to execute an application. A standard application startup looks like this: new ApplicationManager().Run(CreateApplication, RunApplication); [3] "Generic types can trigger an avalanche of generic parameters(tm) throughout your code." The question is: what should the types of the two function parameters be? Does CreateApplication return an IApplication or a caller-specific derived type? What is the type of the application parameter passed to RunApplication? Also IApplication? Or the more derived type returned by CreateApplication? As with the previous example, if the IApplicationManager is to return a derived type, then it must be generic in TApplication and both function parameters will be generically typed as well. These generic types will trigger an avalanche of generic parameters(tm) throughout the other extension methods, interfaces and classes involved in initializing and executing applications. That sounds horrible. This sounds like a pretty easy decision. Why are we even considering the alternative? Well, because it can be very advantageous if the application can declare RunApplication with a strictly typed signature, as shown below. private static void RunApplication(CodeGeneratorApplication application) { ... } Neat, right? I've got my very own type back. [Where Generics Goes off the Rails] However, if the IApplicationManager is to call this function, then the signature of CreateAndStartUp() and Run() have to be generic, as shown below. TApplication CreateAndStartUp( Func createApplication ) where TApplication : IApplication; IApplicationExecutionTranscript Run( Func createApplication, Action run ) where TApplication : IApplication; These are quite messy -- and kinda scary -- signatures. [4] if these core methods are already so complex, any other methods involved in startup and execution would have to be equally complex -- including helper methods created by calling applications. [5] The advantage here is that the caller will always get back the type of application that was created. The compiler guarantees it. The caller is not obliged to cast an IApplication back up to the original type. The disadvantage is that all of the library code is infected by a generic parameter with its attendant IApplication generic constraint. [6] [Don't add Support for Conflicting Patterns] The title of this section seems pretty self-explanatory, but we as designers must remain vigilant against the siren call of what seems like a really elegant and strictly typed solution. "But aren't properties on an application exactly what we just worked so hard to eliminate?" The generics above establish a pattern that must be adhered to by subsequent extenders and implementors. And to what end? So that a caller can attach properties to an application and access those in a statically typed manner, i.e. without casting? But aren't properties on an application exactly what we just worked so hard to eliminate? Isn't the recommended pattern to create a "settings" object and add it to the IOC instead? That is, as of Quino 2.0, you get an IApplication and obtain the desired settings from its IOC. Technically, the cast is still taking place in the IOC somewhere, but that seems somehow less bad than a direct cast. If the framework recommends that users don't add properties to an application -- and ruthlessly eliminated all standard properties and descendants -- then why would the framework turn around and add support -- at considerable cost in maintenance and readability and extendibility -- for callers that expect a certain type of application? [Wrapping up] Let's take a look at the non-generic implementation and see what we lose or gain. The final version of the IApplicationManager API is shown below, which properly balances the concerns of all stakeholders and hopefully will stand the test of time (or at least last until the next major revision). IApplication CreateAndStartUp( Func createApplication ); IApplicationExecutionTranscript Run( Func createApplication, Action run ); These are the hard questions of API design: ensuring consistency, enforcing intent and balancing simplicity and cleanliness of code with expressiveness. -------------------------------------------------------------------------------- [1] A predilection of mine, I'll admit, especially when writing about a topic about which I've thought quite a lot. In those cases, the instinct to just skip "the object" and move on to the esoteric details that stand in the way of an elegant, perfect solution, is very, very strong. [1] This more-realized typing was so attractive that we used it in many places in Quino without properly weighing the consequences. This article is the result of reconsidering that decision. [1] This call looks the same for all UI (console, Winform, WPF, etc.), all services (e.g. ASP.NET, Windows-services, etc.) as well as for automated tests. This fact isn't germane to the discussion above, but it's pretty neat in its own right. All an application has to do is define two methods with the right signatures and call the appropriate Run() method for the desired type of application. Almost all of the startup code is shared and the pattern is the same everywhere. [1] Yes, the C# compiler will allow you to elide generics for most method calls (so long as the compiler can determine the types of the parameters without it). However, generics cannot be removed from constructor calls. These must always specify all generic parameters, which makes for messier-looking, lengthy code in the caller e.g. when creating the ApplicationManager were it to have been defined with generic parameters. Yet another thing to consider when choosing how to define you API. [1] As already mentioned elsewhere (but it bears repeating): callers can, of course, eschew the generic types and use IApplication everywhere -- and most probably will, because the advantage offered by making everything generic is vanishingly small.. If your API looks this scary, entropy will eat it alive before the end of the week, to say nothing of its surviving to the next major version. [1] A more subtle issue that arises is if you do end up -- even accidentally -- mixing generic and non-generic calls (i.e. using IApplication as the extended parameter in some cases and TApplication in others). This issue is in how the application object is registered in the IOC. During development, when the framework was still using generics everywhere (or almost everywhere), some parts of the code were retrieving a reference to the application using the most-derived type whereas the application had been registered in the container as a singleton using IApplication. The call to retrieve the most derived type returned a new instance of the application rather than the pre-registered singleton, which was a subtle and difficult bug to track down. ]]> https://www.earthli.com/news/view_article.php?id=3165 2015-09-19T07:29:59+02:00 https://earthli.com/users/marco

, "part II" and "part III" . The goal of this article is to discuss a concrete example of how we... ]]>

, "part II" and "part III" . The goal of this article is to discuss a concrete example of how we decided whether to use generic type parameters throughout the configuration part of Quino. The meat of that discussion will be in a part 2 because we're going to have to lay some groundwork about the features we want first. (Requirements!) [A Surfeit of Generics] As of "Quino 2.0-beta2" , the configuration library consisted of a central IApplication interface which has a reference to an IOC container and a list of startup and shutdown actions. As shown in "part III" , these actions no longer have a generic TApplication parameter. This makes it not only much easier to use the framework, but also easier to extend it. In this case, we were able to remove the generic parameter without sacrificing any expressiveness or type-safety. As of beta2, there were still several places where generic TApplication parameters were cluttering the API. Could we perhaps optimize further? Throw out even more complexity without losing anything? [Starting up an application] One of these places is the actual engine that executes the startup and shutdown actions. This code is a bit trickier than just a simple loop because Quino supports execution in debug mode -- without exception-handling -- and release mode -- with global exception-handling and logging. As with any application that uses an IOC container, there is a configuration phase, during which the container can be changed and an execution phase, during which the container produces objects but can no longer be re-configured. Until 2.0-beta2, the execution engine was encapsulated in several extension methods called Run(), StartUp() and so on. These methods were generally generic in TApplication. I write "generally" because there were some inconsistencies with extension methods for custom application types like Winform or Console applications. While extension methods can be really useful, this usage was not really appropriate as it violated the open/closed principle. For the final release of Quino, we wanted to move this logic into an IApplicationManager so that applications using Quino could (A) choose their own logic for starting an application and (B) add this startup class to a non-Quino IOC container if they wanted to. [Application Execution Modes] So far, so good. Before we discuss how to rewrite the application manager/execution engine, we should quickly revisit what exactly this engine is supposed to do. As it turns out, not only do we wnat to make an architectural change to make the design more open for extension, but the basic algorithm for starting an application changed, as well. "What does it mean to run an application?" Quino has always acknowledged and kinda/sorta supported the idea that a single application can be run in different ways. Even an execution that results in immediate failure technically counts as an execution, as a traversal of the state machine defined by the application. If we view an application for the state machine that it is, then every application has at least two terminal nodes: OK and Error. But what does OK mean for an application? In Quino, it means that all startup actions were executed without error and the run() action passed in by the caller was also executed without error. Anything else results in an exception and is shunted to Error. [image]But is that true, really? Can you think of other ways in which an application could successfully execute without really having failed? For most applications, the answer is yes. Almost every application -- and certainly every Quino application -- supports a command line. One of the default options for the command line of a Quino application is -h, which shows a manual for the other command-line options. If the application is running in a console, this manual is printed to the console; for a Winform application, a dialog box is shown; and so on. This "help" mode is actually a successful execution of the application that did not result in the main event loop of the application being executed. Thought of in this way, any command-line option that controls application execution could divert the application to another type of terminal node in the state machine. A good example is when an application provides support for importing or exporting data via the command line. ["Canceled" Terminal Nodes] A terminal node is also not necessarily only Crashed or Ok. Almost any application will also need to have a Canceled mode that is a perfectly valid exit state. For example, * If the application requires a login during execution (startup), but the user aborts authentication * If the application supports schema migration, but the user aborts without migrating the schema These are two ways in which a standard Quino application could run to completion without crashing but without having accomplished any of its main tasks. It ran and it didn't crash, but it also didn't do anything useful. [Intermediate Nodes in the Application State Machine] This section title sounds a bit pretentious, but that's exactly what we want to discuss here. Instead of having just start and terminal nodes, the Quino startup supports cycles through intermediate nodes as well. What the hell does that mean? It means that some nodes may trigger Quino to restart in a different mode in order to handle a particular kind of error condition that could be repaired. [1] A concrete example is desperately needed here, I think. The main use of this feature in Quino right now is to support on-the-fly schema-migration without forcing the user to restart the application. This feature has been in Quino from the very beginning and is used almost exclusively by developers during development. The use case to support is as follows: 1. Developer is running an application 2. Developer make change to the model (or pulls changes from the server) 3. Developer runs the application with a schema-change 4. Application displays migration tool; developer can easily migrate the schema and continue working This workflow minimizes the amount of trouble that a developer has when either making changes or when integrating changes from other developers. In all cases in which the application model is different from the developer's database schema, it's very quick and easy to upgrade and continue working. ["Rescuing" an application in Quino 2.0] How does this work internally in Quino 2.0? The application starts up but somehow encounters an error that indicates that a schema migration might be required. This can happen in one of two ways: 1. The schema-verification step in the standard Quino startup detects a change in the application model vis à vis the data schema 2. Some other part of the startup accesses the database and runs into a DatabaseException that is indicative of a schema-mismatch In all of these cases, the application that was running throws an ApplicationRestartException, that the standard IApplicationManager implementation knows how to handle. It handles it by shutting down the running application instance and asking the caller to create a new application, but this time one that knows how to handle the situation that caused the exception. Concretely, the exception includes an IApplicationCreationSettings descendant that the caller can use to decide how to customize the application to handle that situation. The manager then runs this new application to completion (or until a new RestartApplicationException is thrown), shuts it down, and asks the caller to create the original application again, to give it another go. In the example above, if the user has successfully migrated the schema, then the application will start on this second attempt. If not, then the manager enters the cycle again, attempting to repair the situation so that it can get to a terminal node. Naturally, the user can cancel the migration and the application also exits gracefully, with a Canceled state. A few examples of possible application execution paths: * Standard => OK * Standard => Error * Standard => Canceled * Standard => Restart => Migrator => Standard => OK * Standard => Restart => Migrator => Canceled The pattern is the same for interactive, client applications as for headless applications like test suites, which attempt migration once and abort if not successful. Applications like web servers or other services will generally only support the OK and Error states and fail when they encounter a RestartApplicationException. Still, it's nice to know that the pattern is there, should you need it. It fits relatively cleanly into the rest of the API without making it more complicated. The caller passes two functions to the IApplicationManager: one to create an application and one to run it. An example from the Quino CodeGeneratorApplication is shown below: internal static void Main() { new ApplicationManager().Run(CreateApplication, GenerateCode); } private static IApplication CreateApplication( IApplicationCreationSettings applicationCreationSettings ) { ... } private static void GenerateCode(IApplication application) { ... } We'll see in the next post what the final API looks like and how we arrived at the final version of that API in Quino 2.0. -------------------------------------------------------------------------------- [1] Or rescued, using the nomenclature from Eiffel exception-handling, which actually does something very similar. The exception handling in most languages lets you clean up and move on, but the intent isn't necessarily to re-run the code that failed. In Eiffel, this is exactly how exception-handling works: fix whatever was broken and re-run the original code. Quino now works very much like this as well. ]]> https://www.earthli.com/news/view_article.php?id=3164 2015-09-19T07:18:15+02:00 https://earthli.com/users/marco

or "get the latest copy from here" . Chapter 3, Basic Concepts and chapter 4, Best Practices have been included in their entirety below. [3 Best Practices] [3.1 Focused Commits] Focused commits are required; small commits are highly recommended. Keeping the number of changes per commit tightly focused on a single task helps in many cases. * They are easier to resolve when merge conflicts occur * They can be more easily merged/rebased by Git * If a commit addresses only one issue, it is easier for a reviewer or reader to decide whether it should be examined. For example, if you are working on a bug fix and discover that you need to refactor a file as well, or clean up the documentation or formatting, you should finish the bug fix first, commit it and then reformat, document or refactor in a separate commit. Even if you have made a lot of changes all at once, you can still separate changes into multiple commits to keep those commits focused. Git even allows you to split changes from a single file over multiple commits (the Git Gui provides this functionality as does the index editor in SmartGit). [3.2 Snapshots] Use the staging area to make quick snapshots without committing changes but still being able to compare them against more recent changes. For example, suppose you want to refactor the implementation of a class. * Make some changes and run the tests; if everything’s ok, stage those changes * Make more changes; now you can diff these new changes not only against the version in the repository but also against the version in the index (that you staged). * If the new version is broken, you can revert to the staged version or at least more easily figure out where you went wrong (because there are fewer changes to examine than if you had to diff against the original) * If the new version is ok, you can stage it and continue working [3.3 Developing New Code] Where you develop new code depends entirely on the project release plan. * Code for releases should be committed to the release branch (if there is one) or to the develop branch if there is no release branch for that release * If the new code is a larger feature, then use a feature branch. If you are developing a feature in a hotfix or release branch, you can use the optional base parameter to base the feature on that branch instead of the develop branch, which is the default. [3.4 Merging vs. Rebasing] Follow these rules for which command to use to combine two branches: * If both branches have already been pushed, then merge. There is no way around this, as you won’t be able to push a non-merged result back to the origin. * If you work with branches that are part of the standard branching model (e.g. release, feature, etc.), then merge. * If both you and someone else made changes to the same branch (e.g. develop), then rebase. This will be the default behavior during development [4 Development Process] A branching model is required in order to successfully manage a non-trivial project. Whereas a trivial project generally has a single branch and few or no tags, a non-trivial project has a stable release—with tags and possible hotfix branches—as well as a development branch—with possible feature branches. A common branching model in the Git world is called Git Flow. Previous versions of this manual included more specific instructions for using the Git Flow-plugin for Git but experience has shown that a less complex branching model is sufficient and that using standard Git commands is more transparent. However, since Git Flow is a very widely used branching model, retaining the naming conventions helps new developers more easily understand how a repository is organized. [4.1 Branch Types] The following list shows the branch types as well as the naming convention for each type: * master is the main development branch. All other branches should be merged back to this branch (unless the work is to be discarded). Developers may apply commits and create tags directly on this branch. * feature/name is a feature branch. Feature branches are for changes that require multiple commits or coordination between multiple developers. When the feature is completed and stable, it is merged to the master branch after which it should be removed. Multiple simultaneous feature branches are allowed. * release/vX.X.X is a release branch. Although a project can be released (and tagged) directly from the master branch, some projects require a longer stabilization and testing phase before a release is ready. Using a release branch allows development on the develop branch to continue normally without affecting the release candidate. Multiple simultaneous release branches are strongly discouraged. * hotfix/vX.X.X is a hotfix branch. Hotfix branches are always created from the release tag for the version in which the hotfix is required. These branches are generally very short-lived. If a hotfix is needed in a feature or release branch, it can be merged there as well (see the optional arrow in the following diagram). The main difference from the Git Flow branching model is that there is no explicit stable branch. Instead, the last version tag serves the purpose just as well and is less work to maintain. For more information on where to develop code, see “3.3 – Developing New Code”. [4.2 Example] To get a better picture of how these branches are created and merged, the following diagram depicts many of the situations outlined above. The diagram tells the following story: * Development began on the master branch * v1.0 was released directly from the master branch * Development on feature “B” began * A bug was discovered in v1.0 and the v1.0.1 hotfix branch was created to address it * Development on feature “A” began * The bug was fixed, v1.0.1 was released and the fix was merged back to the master branch * Development continued on master as well as features “A” and “B” * Changes from master were merged to feature “A” (optional merge) * Release branch v1.1 was created * Development on feature “A” completed and was merged to the master branch * v1.1 was released (without feature “A”), tagged and merged back to the master branch * Changes from master were merged to feature “B” (optional merge) * Development continued on both the master branch and feature “B” * v1.2 was released (with feature “A”) directly from the master branch [image] Legend: * Circles depict commits * Blue balloons are the first commit in a branch * Grey balloons are a tag * Solid arrows are a required merge * Dashed arrows are an optional merge ]]> https://www.earthli.com/news/view_article.php?id=3161 2015-09-03T12:30:57+02:00 https://earthli.com/users/marco

. The following article provides an update, this time with version 9.2, released just last week. tl;dr: I'm back to ReSharper 8.2.3 and am a bit worried about the state of the 9.x series of ReSharper.... ]]>

. The following article provides an update, this time with version 9.2, released just last week. tl;dr: I'm back to ReSharper 8.2.3 and am a bit worried about the state of the 9.x series of ReSharper. Ordinarily, JetBrains has eliminated performance, stability and functional issues by the first minor version-update (9.1), to say nothing of the second (9.2). [Test Runner] In the previous article, my main gripe was with the unit-test runner, which was unusable due to flakiness in the UI, execution and change-detection. With the release of 9.2, the UI and change-detection problems have been fixed, but the runner is still quite flaky at executing tests. What follows is the text of the report that I sent to JetBrains when they asked me why I uninstalled R# 9.2. As with 9.0 and 9.1, I am unable to productively use the 9.2 Test Runner with many of my NUnit tests. These tests are not straight-up, standard tests, but R# 8.2.3 handled them without any issues whatsoever. What's special about my tests? There are quite a few base classes providing base functionality. The top layers provide scenario-specific input via a generic type parameter. - TestsBase - OtherBase (7 of these, one with an NUnit CategoryAttribute) - ConcreteTests (defines tests with NUnit TestAttributes) - ProviderAConcreteTests (CategoryAttribute) - ProtocolAProviderAConcreteTests (TMixin = ProtocolAProviderA; TestFixtureAttribute, CategoryAttributes) - ProtocolBProviderAConcreteTests (TMixin = ProtocolBProviderA; TestFixtureAttribute, CategoryAttributes) - ProviderBConcreteTests (CategoryAttribute) - ProtocolAProviderBConcreteTests (TMixin = ProtocolAProviderB; TestFixtureAttribute, CategoryAttributes) - ProtocolBProviderBConcreteTests (TMixin = ProtocolBProviderB; TestFixtureAttribute, CategoryAttributes) The test runner in 9.2 is not happy with this at all. The test explorer shows all of the tests correctly, with the test counts correct. If I select a node for all tests for ProviderB and ProtocolA (696 tests in 36 fixtures), R# loads 36 non-expandable nodes into the runner and, after a bit of a wait, marks them all as inconclusive. Running an individual test-fixture node does not magically cause the tests to load or appear and also shows inconclusive (after a while; it seems the fixture setup executes as expected but the results are not displayed). If I select a specific, concrete fixture and add or run those tests, R# loads and executes the runner correctly. If I select multiple test fixtures in the explorer and add them, they also show up as expandable nodes, with the correct test counts, and can be executed individually (per fixture). However, if I elect to run them all by running the parent node, R# once again marks everything as inconclusive. As I mentioned, 8.2.3 handles this correctly and I feel R# 9.2 isn't far off -- the unit-test explorer does, after all, show the correct tests and counts. In 9.2, it's not only inconvenient, but I'm worried that my tests are not being executed with the expected configuration. Also, I really missed the StyleCop plugin for 9.2. There's a beta version for 9.1 that caused noticeable lag, so I'm still waiting for a more unobtrusive version for 9.2 (or any version at all). While it's possible that there's something I'm doing wrong, or there's something in my installation that's strange, I don't think that's the problem. As I mentioned, test-running for the exact same solution with 8.2.3 is error-free and a pleasure to use. In 9.2, the test explorer shows all of the tests correctly, so R# is clearly able to interpret the hierarchy and attributes (noted above) as I've intended them to be interpreted. This feels very much like a bug or a regression for which JetBrains doesn't have test coverage. I will try to work with them to help them get coverage for this case. [Real-Time StyleCop rules] Additionally, the StyleCop plugin is absolutely essential for my workflow and there still isn't an official release for any of the 9.x versions. ReSharper 9.2 isn't supported at all yet, even in prerelease form. The "official Codeplex page" shows the latest official version as 4.7, released in January of 2012 for ReSharper 8.2 and Visual Studio 2013. One would imagine that VS2015 support is in the works, but it's hard to say. There is a "page for StyleCop in the ReSharper extensions gallery" but that shows a beta4, released in April of 2015, that only works with ReSharper 9.1.x, not 9.2. I tested it with 9.1.x, but it noticeably slowed down the UI. While typing was mostly unaffected, scrolling and switching file-tabs was very laggy. Since StyleCop is essential for so many developers, it's hard to see why the plugin gets so little love from either JetBrains or Microsoft. [GoTo Word] The ""Go To Word" plugin" is not essential but it is an extremely welcome addition, especially with so much more client-side work depending on text-based bindings that aren't always detected by ReSharper. In those cases, you can find -- for example -- all the references of a "Knockout template" by searching just as you would for a type or member. Additionally, you benefit from the speed of the ReSharper indexing engine and search UI instead of using the comparatively slow and ugly "Find in Files" support in Visual Studio. Alternatives suggested in the comments to the linked issue above all depend on building yet another index of data (e.g. "Sando Code Search Tool" ). JetBrains has pushed off integrating go-to-word until version 10. Again, not a deal-breaker, but a shame nonetheless, as I'll have to do without it in 9.x until version 10 is released. With so much more client-side development going on in Visual Studio and with dynamic languages and data-binding languages that use name-matching for data-binding, GoToWord is more and more essential. Sure, ReSharper can continue to integrate native support for finding such references, but until that happens, we're stuck with the inferior Find-in-Files dialog or other extensions that increase the memory pressure for larger solutions. ]]> https://www.earthli.com/news/view_article.php?id=3142 2015-05-30T23:51:19+02:00 https://earthli.com/users/marco

is also available for those with access to the Encodo issue tracker. [Highlights] In "beta1" , we read about changes to configuration, the data driver architecture, DDL commands, and security... ]]>

is also available for those with access to the Encodo issue tracker. [Highlights] In "beta1" , we read about changes to configuration, the data driver architecture, DDL commands, and security and access control in web applications. In beta-2, we made the following additional improvements: * Introduced a new generated-code version that avoids all global references and provides a much-improved API for working with metadata. See below for more details. ("QNO-4821" , "QNO-4820" , "QNO-4819" , "QNO-4818" , "QNO-4817" , "QNO-4816" , "QNO-4815" , "QNO-4814" , "QNO-4813" , "QNO-4812" , "QNO-4507" , "QNO-4506" , "QNO-4117" ) * Continued to update and standardize configuration and execution for applications, as documented in Encodo's configuration library for Quino: "part I" , "part II" and "part III" . ("QNO-4834" , "QNO-4831" , "QNO-4809" , "QNO-4772" , "QNO-4663" , "QNO-4664" , "QNO-4360" , "QNO-4771" , "QNO-4676" , "QNO-4675" , "QNO-4673" , "QNO-3765" ) * Moved all globals and static classes with global dependencies to the application's IOC container. ("QNO-4791" , "QNO-4792" , "QNO-4790" , "QNO-4784" , "QNO-4782" , "QNO-4778" , "QNO-4787" , "QNO-4774" ) * Reduce direct dependencies and properties in IApplication, ICoreApplication and IMetaApplication. ("QNO-4789" , "QNO-4788" , "QNO-4786" , "QNO-4785" , "QNO-4671" , "QNO-4669" , , "QNO-4668" , "QNO-4667" , "QNO-4660" ) * Removed references to GlobalContext and ServiceLocator from the Winforms components ("QNO-4832" ) * Standardized naming and namespaces to conform to industry, StyleCop and .NET conventions. ("QNO-4807" , "QNO-4806" ) * Added support for parameterized custom SQL queries with the ICustomCommandBuilder. This was added by customer request, for applications that formulate queries that are beyond what the Quino ORM is currently capable of mapping. A blog post with more detail on how this works is forthcoming. ("QNO-4802" ) * Further cleanup and consolidation in the data-driver hierarchy. This work was a direct result of Daniel Roth's Bachelor's thesis work in which he integrated NHibernate as an alternative ORM for Quino. ("QNO-4808" ; still to do by RTM: "QNO-4749" ) * Discontinued support for DataContract and DataMember attributes in metadata and generated code. ("QNO-4823" , "QNO-4826" ) [Goodbye, old friends] This release addressed some issues that have been bugging us for a while (almost 3 years in one case). * "QNO-3765" (32 months): After a schema migration caused by a DatabaseException on login, restart the application * "QNO-4117" (27 months): PreferredType registration for models is not always executed * "QNO-4408" (18 months): When access to the remoting server is unauthorized, the web site should respond with an error * "QNO-4506" (14 months): The code generator should generate the persistent object and metadata references in separate classes * "QNO-4507" (14 months): Business objects for modules should not rely on GlobalContext in generated code You will not be missed. [Breaking changes] As we've mentioned before, this release is absolutely merciless in regard to backwards compatibility. Old code is not retained as obsolete Obsolete. Instead, a project upgrading to 2.0 will encounter compile errors. That said, if you arm yourself with a bit of time, ReSharper and the release notes (and possibly keep an Encodo employee on speed-dial), the upgrade is not difficult. It consists mainly of letting ReSharper update namespace references for you. In cases where the update is not so straightforward, we've provided release notes. [V1 generated code support] One of the few things you'll be able to keep (at least for a minor version or two) is the old-style generated code. We made this concession because, while even a large solution can be upgraded from 1.13.0 to 2.0 relatively painlessly in about an hour (we've converted our own internal projects to test), changing the generated-code format is potentially a much larger change. Again, an upgrade to the generated-code format isn't complicated but it might require more than an hour or two's worth of elbow grease to complete. Therefore, you'll be able to not only retain your old generated code, but the code generator will continue support the old-style code-generation format for further development. Expect the grace period to be relatively short, though. Regardless of whether you elect to keep the old-style generated code, you'll have to do a little bit of extra work just to be able to generate code again. 1. Manually update a couple of generated files, as shown below. 2. Compile the solution 3. Generate code with the Quino tools Before you can regenerate, you'll have to manually update your previously generated code in the main model file, as shown below. [Previous version] static MyModel() { Messages = new InMemoryRecorder(); Loader = new ModelLoader(() => Instance, () => Messages, new MyModelGenerator()); } public static IMetaModel CreateModel(IExtendedRecorder recorder) { if (recorder == null) { throw new ArgumentNullException("recorder"); } var result = Loader.Generator.CreateModel(recorder); result.Configure(); return result; } // More code ... /// protected override void DoConfigure() { base.DoConfigure(); ConfigurePreferredTypes(); ApplyCustomConfiguration(); } [Manually updated version] static MyModel() { Messages = new InMemoryRecorder(); Loader = new ModelLoader(() => Instance, () => Messages, new MyModelGenerator()); } public static IMetaModel CreateModel(IExtendedRecorder recorder) { if (recorder == null) { throw new ArgumentNullException("recorder"); } var result = Loader.Generator(MyModel)new MyModelGenerator().CreateModel( ServiceLocator.Current.GetInstance(), ServiceLocator.Current.GetInstance(), recorder ); result.ConfigurePreferredTypes(); result.ApplyCustomConfiguration(); return result; } /// protected override void DoConfigure() { base.DoConfigure(); ConfigurePreferredTypes(); ApplyCustomConfiguration(); } [Integrate into the model builder] In the application configuration for the first time you generate code with Quino 2.0, you should use: ModelLoader = MyModel.Loader; this.UseMetaSimpleInjector(); this.UseModelLoader(MyModel.CreateModel); After regenerating code, you should use the following for version-2 generated code: ModelLoader = MyModel.Loader; this.UseMetaSimpleInjector(); this.UseModelLoader(MyModelExtensions.CreateModelAndMetadata); ...and the following for version-1 generated code: ModelLoader = MyModel.Loader; this.UseMetaSimpleInjector(); this.UseModelLoader(MyModel.CreateModel); [Still to do by RTM] As you can see, we've already done quite a bit of work in beta1 and beta2. We have a few more tasks planned for the feature-complete release candidate for 2.0 Move the schema-migration metadata table to a module. The Quino schema-migration extracts most of the information it needs from database schema itself. It also stores extra metadata in a special table. This table has been with Quino since before modules were supported (over seven years) and hence was built in a completely custom manner. Moving this support to a Quino metadata module will remove unnecessary implementation and make the migration process more straightforward. ("QNO-4888" ) Separate collection algorithm from storage/display method in IRecorder and descendants. The recording/logging library has a very good interface but the implementation for the standard recorders has become too complex as we added support for multi-threading, custom disposal and so on. We want to clean this up to make it easier to extend the library with custom loggers. ("QNO-4888" ) Split up Encodo and Quino assemblies based on functionality. There are only a very dependencies left to untangle ("QNO-4678" , "QNO-4672" , "QNO-4670" ); after that, we'll split up the two main Encodo and Quino assemblies along functional lines. ("QNO-4376" ) Finish integrating building and publishing NuGet and symbol packages into Quino's release process. And, finally, once we have the assemblies split up to our liking, we'll finalize the Nuget packages for the Quino library and leave the direct-assembly-reference days behind us, ready for Visual Studio 2015. ("QNO-4376" ) That's all we've got for now. See you next month for the next (and, hopefully, final update)! ]]> https://www.earthli.com/news/view_article.php?id=3137 2015-05-17T17:45:56+02:00 https://earthli.com/users/marco

discusses the history of the configuration system in Quino as well as a handful of principles we kept in mind while designing the new system 2. "part II" discusses the basic architectural changes and compares an example from the old ... ]]>

discusses the history of the configuration system in Quino as well as a handful of principles we kept in mind while designing the new system 2. "part II" discusses the basic architectural changes and compares an example from the old configuration system to the new. 3. part III takes a look at configuring the "execution order" -- the actions to execute during application startup and shutdown [Introduction] Registering with an IOC is all well and good, but something has to make calls into the IOC to get the ball rolling. "Something has to actually make calls into the IOC to get the ball rolling." Even service applications -- which start up quickly and wait for requests to do most of their work -- have basic operations to execute before declaring themselves ready. Things can get complex when starting up registered components and performing basic checks and non-IOC configuration. * In which order are the components and configuration elements executed? * How do you indicate dependencies? * How can an application replace a piece of the standard startup? * What kind of startup components are there? Part of the complexity of configuration and startup is that developers quickly forget all of the things that they've come to expect from a mature product and start from zero again with each application. Encodo and Quino applications take advantage of prior work to include standard behavior for a lot of common situations. [Configuration Patterns] Some components can be configured once and directly by calling a method like UseMetaTranslations(string filePath), which includes all of the configuration options directly in the composition call. This pattern is perfect for options that are used only by one action or that it wouldn't make sense to override in a subsequent action. So, for simple actions, an application can just replace the existing action with its own, custom action. In the example above, an application for which translations had already been configured would just call UseMetaTranslations() again in order to override that behavior with its own. "Most application will replace standard actions or customize standard settings" Some components, however, will want to expose settings that can be customized by actions before they are used to initialize the component. For example, there is an action called SetUpLoggingAction, which configures logging for the application. This action uses IFileLogSettings and IEventLogSettings objects from the IOC during execution to determine which types of logging to configure. An application is, of course, free to replace the entire SetUpLoggingAction action with its own, completely custom behavior. However, an application that just wanted to change the log-file behavior or turn on event-logging could use the Configure() method [1], as shown below. application.Configure( s => s.Behavior = LogFileBehavior.MultipleFiles ); application.Configure( s => s.Enabled = true ); [Actions] A Quino application object has a list of StartupActions and a list of ShutdownActions. Most standard middleware methods register objects with the IOC and add one or more actions to configure those objects during application startup. Actions have existed for quite a while in Quino. In Quino 2, they have been considerably simplified and streamlined to the point where all but a handful are little more than a functional interface [2]. The list below will give you an idea of the kind of configuration actions we're talking about. * Load configuration data * Process command line * Set up logging * Upgrade settings/configuration (e.g. silent upgrade) * Log a header (e.g. user/date/file locations/etc.; for console apps. this might be mirrored to the console) * Load plugins * Set up standard locations (e.g. file-system locations) For installed/desktop/mobile applications, there's also: * Initialize UI components * Provide loading feedback * Check/manage multiple running instances * Check software update * Login/authentication Quino applications also have actions to configure metadata: * Configure expression engine * Load metadata * Load metadata-overlays * Validate metadata * Check data-provider connections * Check/migrate schema * Generate default data Application shutdown has a smaller set of vital cleanup chores that: * dispose of connection managers and other open resources * write out to the log, flush it and close it * show final feedback to the user [Anatomy of an Action] The following example [3] is for the 1.x version of the relatively simple ConfigureDisplayLanguageAction. public class ConfigureDisplayLanguageAction : ApplicationActionBase where TApplication : ICoreApplication { public ConfigureDisplayLanguageAction() : base(CoreActionNames.ConfigureDisplayLanguage) { } protected override int DoExecute( TApplication application, ConfigurationOptions options, int currentResult) { // Configuration code... } } What is wrong with this startup action? The following list illustrates the main points, each of which is addressed in more detail in its own section further below. * The ConfigurationOptions parameter introduces an unnecessary layer of complexity * The generic parameter TApplication complicates declaration, instantiation and extension methods that use the action * The int return type along with the currentResult parameter are a bad way of controlling flow. The same startup action in Quino 2.x has the following changes from the Quino 1.x version above (legend: additions; deletions). public class ConfigureDisplayLanguageAction : ApplicationActionBase where TApplication : ICoreApplication { public ConfigureDisplayLanguageAction() : base(CoreActionNames.ConfigureDisplayLanguage) { } publicprotected override void int DoExecute( TApplication application, ConfigurationOptions options, int currentResult) { // Configuration code... } } As you can see, quite a bit of code and declaration text was removed, all without sacrificing any functionality. The final form is quite simple, inheriting from a simple base class that manages the name of the action and overrides a single parameter-less method. It is now much easier to see what an action does and the barrier to entry for customization is much lower. public class ConfigureDisplayLanguageAction : ApplicationActionBase { public ConfigureDisplayLanguageAction() : base(CoreActionNames.ConfigureDisplayLanguage) { } public override void Execute() { // Configuration code... } } In the following sections, we'll take a look at each of the problems indicated above in more detail. [Remove the ConfigurationOptions parameter] These options are a simple enumeration with values like Client, Testing, Service and so on. They were used only by a handful of standard actions. These options made it more difficult to decide how to implement the action for a given task. If two tasks were completely different, then a developer would know to create two separate actions. However, if two tasks were similar, but could be executed differently depending on application type (e.g. testing vs. client), then the developer could still have used two separate actions, but could also have used the configuration options. Multiple ways of doing the exact same thing is all kinds of bad. "Multiple ways of doing the exact same thing is all kinds of bad." Parameters like this conflict conceptually with the idea of using composition to build an application. To keep things simple, Quino applications should be configured exclusively by composition. Composing an application with service registrations and startup actions and then passing options to the startup introduced an unneeded level of complexity. Instead, an application now defines a separate action for each set of options. For example, most applications will need to set up the display language to use -- be it for a GUI, a command-line or just to log messages in the correct language. For that, the application can add a ConfigureDisplayLanguageAction to the startup actions or call the standard method UseCore(). Desktop or single-user applications can use the ConfigureGlobalDisplayLanguageAction or call UseGlobalCore() to make sure that global language resources are also configured. [Remove the TApplication generic parameter] The generic parameter to this interface complicates the IApplication interface and causes no end of trouble in MetaApplication, which actually inherits from IApplication for historical reasons. "There is no need to maintain statelessness for a single-use object." Originally, this parameter guaranteed that an action could be stateless. However, each action object is attached to exactly one application (in the IApplication.StartupActions list. So the action that is attached to an application is technically stateless, and a completely different application than the one to which the action is attached could be passed to the IApplcationAction.Execute...which makes no sense whatsoever. Luckily, this never happens, and only the application to which the action is attached is passed to that method. If that's the case, though, why not just create the action with the application as a constructor parameter when the action is added to the StartupActions list? There is no need to maintain statelessness for a single-use object. This way, there is no generic parameter for the IApplication interface, all of the extension methods are much simpler and applications are free to create custom actions that work with descendants of IApplication simply by requiring that type in the constructor parameter. [Debugging is important] "A global exception handler is terrible for debugging" The original startup avoided exceptions, preferring an integer return result instead. In release mode, a global exception handler is active and is there to help the application exit more or less smoothly -- e.g. by logging the error, closing resources where possible, and so on. A global exception handler is terrible for debugging, though. For exceptions that are caught, the default behavior of the debugger is to stop where the exception is caught rather than where it is thrown. Instead, you want exceptions raised by your application to to stop the debugger from where they are thrown. So that's part of the reason why the startup and shutdown in 1.x used return codes rather than exceptions. [Multiple valid code paths] The other reason Quino used result codes is that most non-trivial applications actually have multiple paths through which they could successfully run. Exactly which path the application should take depends on startup conditions, parameters and so on. Some common examples are: * Show command-line help * Migrate an application schema * Import, export or generate data To show command-line help, an application execute its startup actions in order. It reaches the action that checks whether the user requested command-line help. This action processes the request, displays that help and then wants to smoothly exit the application. The "main" path -- perhaps showing the user a desktop application -- should no longer be executed. "Non-trivial applications have multiple valid run profiles." Similarly, the action that checks the database schema determines that the schema in the data provider doesn't match the model. In this case, it would like to offer the user (usually a developer) the option to update the schema. Once the schema is updated, though, startup should be restarted from the beginning, trying again to run the main path. [Use exceptions to indicate errors] Whereas the Quino 1.x startup addressed the design requirements above with return codes, this imposes an undue burden on implementors. There was also confusion as to when it was OK to actually throw an exception rather than returning a special code. Instead, the Quino 2.x startup always uses exceptions to indicate errors. There are a few special types of exceptions recognized by the startup code that can indicate whether the application should silently -- and successfully -- exit or whether the startup should be attempted again. [Conclusion] There is of course more detail into which we could go on much of what we discussed in these three articles, but that should suffice for an overview of the Quino configuration library. -------------------------------------------------------------------------------- [1] If C# had them, that it is. See "Java 8" for an explanation of what they are. [1] This pattern is echoed in the latest beta of the ASP.NET libraries, as described in the article "Strongly typed routing for ASP.NET MVC 6 with IApplicationModelConvention" . [1] Please note that formatting for the code examples has been adjusted to reduce horizontal space. The formatting does not conform to the "Encodo C# Handbook" . ]]> https://www.earthli.com/news/view_article.php?id=3138 2015-05-17T17:45:20+02:00 https://earthli.com/users/marco

. We wrapped up that part with the following principles to keep in mind while designing the new system. * Consistency * Opt-in configuration * Inversion of Control * Configuration vs. Execution * Common Usage [Borrowing] ... ]]>

. We wrapped up that part with the following principles to keep in mind while designing the new system. * Consistency * Opt-in configuration * Inversion of Control * Configuration vs. Execution * Common Usage [Borrowing from ASP.NET vNext] Quino's configuration inconsistencies and issues have been well-known for several versions -- and years -- but the opportunity to rewrite it comes only now with a major-version break. Luckily for us, ASP.NET has been going through a similar struggle and evolution. We were able to model some of our terminology on the patterns from their next version. For example, ASP.NET has moved to a pattern where an application-builder object is passed to user code for configuration. The pattern there is to include middleware (what we call "configuration") by calling extension methods starting with "Use". Quino has had a similar pattern for a while, but the method names varied: "Integrate", "Add", "Include"; these methods have now all been standardized to "Use" to match the prevailing .NET winds. [Begone configuration and feedback] Additionally, Quino used to make a distinction between an application instance and its "configuration" -- the template on which an application is based. No more. Too complicated. This design decision, coupled with the promotion of a platform-specific "Feedback" object to first-level citizen, led to an explosion of generic type parameters. [1] The distinction between configuration (template) and application (instance) has been removed. Instead, there is just an application object to configure. The feedback object is now to be found in the service locator. An application registers a platform-specific feedback to use as it would any other customization. [1] The CustomWinformFeedback in the Quino 1.x code at the end of this article provides a glaring example. [Hello service locator] ASP.NET vNext has made the service locator a first-class citizen. In ASP.NET, applications receive an IApplicationBuilder in one magic "Configure" method and receive an IServiceCollection in another magic "ConfigureServices" method. In Quino 2.x, the application is in charge of creating the service container, though Quino provides a method to create and configure a standard one (SimpleInjector). That service locator is passed to the IApplication object and subsequently accessible there. Services can of course be registered directly or by calling pre-packaged Middleware methods. Unlike ASP.NET vNext, Quino 2.x makes no distinction between configuring middleware and including the services required by that middleware. [Begone configuration hierarchy] Quino's configuration library has its roots in a time before we were using an IOC container. The configuration was defined as a hierarchy of configuration classes that modeled the following layers. * A base implementation that makes only the most primitive assumptions about an application. For example, that it has a RunMode ("debug" or "release") or an exit code or that it has a logging mechanism (e.g. IRecorder). * The "Core" layer comprises application components that are very common, but do not depend on Quino's metadata. * And, finally, the "Meta" layer includes configuration for application components that extend the core with metadata-dependent versions as well as specific components required by Quino applications. While these layers are still somewhat evident, the move to middleware packages has blurred the distinction between them. Instead of choosing a concrete configuration base class, an application now calls a handful of "Use" methods to indicate what kind of application to build. There are, of course, still helpful top-level methods -- e.g. UseCore() and UseMeta() methods -- that pull in all of the middleware for the standard application types. But, crucially, the application is free to tweak this configuration with more granular calls to register custom configuration in the service locator. This is a flexible and transparent improvement over passing esoteric parameters to monolithic configuration methods, as in the previous version. [An example: Configure a software updater] Just as a simple example, whereas a Quino 1.x standalone application would set ICoreConfiguration.UseSoftwareUpdater to true, a Quino 2.x application calls UseSoftwareUpdater(). Where a Quino 1.x Winform application would inherit from the WinformFeedback in order to return a customized ISoftwareUpdateFeedback, a Quino 2.x application calls UseSoftwareUpdateFeedback(). The software-update feedback class is defined below and is used by both versions. public class CustomSoftwareUpdateFeedback : WinformSoftwareUpdateFeedback { protected override ResponseType DoConfirmUpdate(TApplication application, ...) { ... } } That's where the similarities end, though. The code samples below show the stark difference between the old and new configuration systems. [Quino 1.x] As explained above, Quino 1.x did not allow registration of a sub-feedback like the software-updater. Instead, the application had to inherit from the main feedback and override a method to create the desired sub-feedback. class CustomWinformFeedback : WinformFeedback { public virtual ISoftwareUpdateFeedback GetSoftwareUpdateFeedback() where TApplication : ICoreApplication where TConfiguration : ICoreConfiguration where TFeedback : ICoreFeedback { return new CustomSoftwareUpdateFeedback(this); } } var configuration = new CustomConfiguration() { UseSoftwareUpdater = true } WinformDxMetaConfigurationTools.Run( configuration, app => new CustomMainForm(app), new CustomWinformFeedback() ); The method-override in the feedback was hideous and scared off a good many developers. not only that, the pattern was to use a magical, platform-specific WinformDxMetaConfigurationTools.Run method to create an application, run it and dispose it. [Quino 2.x] Software-update feedback-registration in Quino 2.x adheres to the principles outlined at the top of the article: it is consistent and uses common patterns (functionality is included and customized with methods named "Use"), configuration is opt-in, and the IOC container is used throughout (albeit implicitly with these higher-level configuration methods). using (var application = new CustomApplication()) { application.UseMetaWinformDx(); application.UseSoftwareUpdater(); application.UseSoftwareUpdaterFeedback(new CustomSoftwareUpdateFeedback()); application.Run(app => new CustomMainForm(app)); } Additionally, the program has complete control over creation, running and disposal of the application. No more magic and implicit after-the-fact configuration. [What comes after configuration?] In the next and (hopefully) final article, we'll take a look at configuring execution -- the actions to execute during startup and shutdown. Registering objects in a service locator is all well and good, but calls into the service locator have to be made in order for anything to actually happen. Keeping this system flexible and addressing standard application requirements is a challenging but not insurmountable problem. Stay tuned. ]]> https://www.earthli.com/news/view_article.php?id=3132 2015-04-10T15:36:06+02:00 https://earthli.com/users/marco

. With beta2 development underway, I thought I'd share some more of the thought process behind the forthcoming changes. [Software Libraries] "what sort of patterns" ... ]]>

. With beta2 development underway, I thought I'd share some more of the thought process behind the forthcoming changes. [Software Libraries] "what sort of patterns integrate and customize the functionality of libraries in an application?" An application comprises multiple tasks, only some of which are part of that application's actual domain. For those parts not in the application domain, software developers use libraries. A library captures a pattern or a particular way of doing something, making it available through an abstraction. These simplify and smooth away detail irrelevant to the application. A runtime and its standard libraries provide many such abstractions: for reading/writing files, connecting to networks and so on. Third-party libraries provide others, like logging, IOC, task-scheduling and more. Because Encodo's been writing software for a long time, we have a lot of patterns that we've come up with for our applications. These libraries are split into two main groups: * Encodo.*: extensions to the .NET framework or third-party libraries that don't depend on Quino metadata. * Quino.*: extensions to the .NET framework, third-party libraries or Encodo libraries that depend on Quino metadata. A sort of "meta" library that lies on top of all of this is configuration and startup of applications that use these libraries. That is, what sort of patterns integrate and customize the functionality of libraries in an application? [Balancing K.I.S.S. and D.R.Y] Almost nowhere in an application is the balance between K.I.S.S. and D.R.Y. more difficult to maintain than in configuration and startup. "So if we already know all of that, why does Quino need a new configuration library?" As mentioned above, there is a lot of commonality between applications in this area. An application will definitely want to incorporate such common configuration from a library. Updates and improvements to that library will then be applied as for any other. This is a good thing. However, an application will also want to be able to tweak almost any given facet of this shared configuration. That is: just keep the good parts, have those upgraded when they're changed, but apply customization and extend functionality for the application's domain. Easy, right? It is here that a good configuration library will find just the right level of granularity for customization. Too coarse? Then an application ends up throwing out too much common configuration in order to customize a small part of it. Too fine? Then the configuration system is too verbose or complex and the application avoids using it. Instead, a configuration system should establish clear patterns -- optimally, just one -- for how to apply customization. * The builder of the underlying configuration library has to consider the myriad situations that might face a library developer and distill those requirements to a common pattern. * The library developer needs to think about which parts an application might want to customize and think about how to expose them. So if we already know all of that, then why does Quino need a new configuration library? Well... [History of Quino's Configuration Library] It's really easy to make things over-complicated and muddy. It's really easy to end up growing several different kinds of extension systems over the years. Quino ended up with a generics-heavy API that made declaring new configuration components very wordy. The core of Quino is the metadata definition for an application domain. That part has barely changed at all since we first wrote it lo so many years ago. We declared it to be our core business -- the part that we are better than others at -- the part we wanted to have under our own control. Our first draft [1] has held up remarkably well. Many of the other components have undergone quite a bit of flux: changes in requirements and the components themselves as well as new development processes and patterns all contributed to change. Over time, various applications had different needs and made adjustments to a different iteration of the configuration library. We moved from supporting only single-threaded, single-user desktop applications to also supporting multi-user, multi-threaded services and web servers. "...we were left with an ugly configuration system that no-one wanted to extend or use -- so yet another would be invented." For all of these different applications, we naturally wanted to maintain the common configuration where possible -- but customizations for new platforms stretched the capabilities of the configuration library. Customization would be made to a new version of that library, but applications that couldn't be upgraded immediately forced backwards-compatibility and thus resulted in several different concurrent ways of configuring a particular facet of an application. In order to keep things in one place, we ended up breaking the interface-separation rule. Dependencies started clumping drastically, but it was OK because nobody was trying to use one thing without the other ten. But it was hard to see what was going on; customization became a black box for all but one or two gurus. On and on it went, until we were left with an ugly configuration system that no-one wanted to extend or use -- so yet another would be invented, ad-hoc. And so it went. [Principles for Quino 2.0 Configuration] With Quino 2.0, we examined the existing system and came up with a list of principles. * Consistency: there should be only be one way of customizing settings and components. When a developer asks how to change something, the answer should always be the same pattern. If not, there better be a damned good reason (see "Configuration vs. Execution" below). * Opt-in configuration: No more magic methods or base classes that automatically add components and settings in black boxes. Even if the application has to call one or two more methods, it's better to be declarative than clever(tm). * Inversion of Control: Standardize configuration to use an IOC container or service locator wherever possible. Instead of clumping settings in configuration or application objects, create discrete settings and put them in the container. Make dependencies explicit (constructor parameters!) and resolved through the container wherever possible. * Configuration vs. Execution: Be very aware of the difference between the "configuration" phase and the "execution" phase. During configuration, the service locator is used in write-only mode; during execution, the service locator is in read-only mode. Code executed during configuration must rely only on explicit dependency-injection via constructor. * Common Usage: Establish a pattern for calling configuration methods, from least to most specific. E.g. call Quino's base configuration methods before any application-specific customization. Establish patterns for how to configure a single startup action or how to create settings for a larger component that could be further customized in subsequent phases. In the next part, we'll take a look at some concrete examples and documentation for the new patterns. [2] -------------------------------------------------------------------------------- [1] To be fair, it wasn't our first attempt at metadata. In one way or another, we'd been defining metadata structures for generic programming for more years than we'd be comfortable divulging. A h/t of course to Opus Software's Atlas libraries -- 1 and 2 -- where many of us contributed. Also, I had experience with cross-platform, generic libraries in C++ stretching all the way back to the late 90s as well as the generalized/meta elements of the "earthli WebCore" . So it was more like the fourth or fifth shot at it, if we're going to be honest -- but at least we got it right. :-) [1] In particular, I'll add more detail about "Common Usage" for those who might feel I've left them hanging a bit in the last bullet point. Sorry 'bout that. The day is only so long. See you next time... ]]> https://www.earthli.com/news/view_article.php?id=3128 2015-03-28T23:26:29+01:00 https://earthli.com/users/marco

is also available for those with access to the Encodo issue tracker. [Highlights] These are the big ones that forced a major-version change. * Rewrote configuration and application API. ("QNO-4666" ... ]]>

is also available for those with access to the Encodo issue tracker. [Highlights] These are the big ones that forced a major-version change. * Rewrote configuration and application API. ("QNO-4666" , "QNO-4679" , "QNO-4659" , "QNO-4772" , "QNO-4663" , "QNO-4664" , "QNO-4360" ) * Data-driver architecture has been consolidated. All drivers, ADO, Mongo and Remote, now use the same base implementation, logic and optimization. ("QNO-4461" , "QNO-2913" , "QNO-4683" ) * Rewrote schema migration to return a list of DDL commands ("QNO-4732" , "QNO-4726" , "QNO-4727" ) Also improved schema-migration and mapping to database ("QNO-4708" , "QNO-4709" , "QNO-4605" , "QNO-4725" , "QNO-4605" , "QNO-4728" , "QNO-4720" , "QNO-4728" ) * Updated the entire security and access-control and authentication API, including adding extensive support for tokens for both ASP.NET MVC and WebAPI. ("QNO-4754" , "QNO-4757" , "QNO-4747" >) * Renamed IMessageRecorder to IRecorder, IMessageStore to IInMemoryRecorder and consolidated IFilteredMessageRecorder to IRecorder. ("QNO-4686" , "QNO-4696" , "QNO-4750" , "QNO-4557" ) Some smaller, but important changes: * Added support for RunInTransaction attribute. Specify the attribute on any IMetaTestFixture to wrap a test or every test in a fixture in a transaction. ("QNO-4682" ) * Shared connection manager is now disposed when an application is disposed. ("QNO-4752" ) [Breaking changes] Oh yeah. You betcha. This is a major release and we've knowingly made a decision not to maintain backwards-compatibility at all costs. Good news, though, the changes to make are relatively straightforward and easy to make if you've got a tool like ReSharper that can update using statements automatically. [Namespace changes] As we saw in "part I" and "part II" of the guide to using NDepend, Quino 2.0 has unsnarled quite a few dependency issues. A large number of classes and interfaces have been moved out of the Encodo.Tools namespace. Many have been moved to Encodo.Core but others have been scattered into more appropriate and more specific namespaces. This is one part of the larger changes, easily addressed by using ReSharper to Alt + Enter your way through the compile errors. [Logging changes] Another large change is in renaming IMessageRecorder to IRecorder and IMessageStore to IInMemoryRecorder. Judicious use of search/replace or just a bit of elbow grease will get you through these as well. [Configuration changes] Finally, probably the most far-reaching change is in merging IConfiguration into IApplication. In previous versions of Quino, applications would create a configuration object and pass that to a platform-dependent Quino Run() method. Some configuration was provided by the application and some by the platform-specific method. The example for Quino 1.13.0 below comes from the JobVortex Winform application. var configuration = new JobVortexConfiguration { MainSettings = Settings.Default }; configuration.Add(new JobVortexClientConfigurationPackage()); if (!string.IsNullOrEmpty(Settings.Default.DisplayLanguage)) { configuration.DisplayLanguage = new Language(Settings.Default.DisplayLanguage); } WinformDxMetaConfigurationTools.Run( configuration, app => new MainForm(app) ); In Quino 2.0, the code above has been rewritten as shown below. using (IMetaApplication application = new JobVortexApplication()) { application.MainSettings = Settings.Default; application.UseJobVortexClient(); if (!string.IsNullOrEmpty(Settings.Default.DisplayLanguage)) { application.DisplayLanguage = new Language(Settings.Default.DisplayLanguage); } application.Run(app => new MainForm(app)); } As you can see, instead of creating a configuration, the program creates an application object. Instead of using configuration packages mixed with extension methods named "Integrate", "Configure" and so on, the new API uses "Use" everywhere. This should be comfortable for people familiar with the OWIN/Katana configuration pattern. It does, however, mean that the IConfiguration, ICoreConfiguration and IMetaConfiguration don't exist anymore. Instead, use IApplication, ICoreApplication and IMetaApplication Again, a bit of elbow grease will be needed to get through these compile errors, but there's little to no risk or need for high-level decisions. There are a lot of these prepackaged methods to help you create common kinds of applications: * UseCoreConsole() (a non-Quino application that uses the console) * UseMetaConsole() (a Quino application that uses the console) * UseCoreWinformDx() (a non-Quino application that uses Winform) * UseMetaWinformDx() (a Quino application that uses Winform) * UseReporting() * UseRemotingServer() * Etc. I think you get the idea. Once we have a final release for Quino 2.0, we'll write more about how to use this new pattern. [Looking ahead to 2.0 Final] This is still just an internal beta of the 2.0 final version. More changes are on the way, including but not limited to: * Remove IConfigurationPackage and standardize the configuration API to be named "Use" everywhere ("QNO-4771" ) * GenericObject improvements ("QNO-4761" , "QNO-4762" ) * Change compile location for all projects ("QNO-4756" ) * Move a lot of properties from ICoreApplication and IMetaApplication to configuration objects in the service locator. Also improve use of and configuration of service locator ("QNO-4659" ) * More improvements to the recorders and logging ("QNO-4688" ) * Changes to how ORM objects and metadata are generated. ("QNO-4506" ) * Separate Encodo and Quino assemblies into multiple, smaller assemblies ("QNO-4376" ) See you there! ]]> https://www.earthli.com/news/view_article.php?id=3125 2015-03-13T08:59:09+01:00 https://earthli.com/users/marco

. [C# 6 Recap] You may be surprised at the version number "7" -- aren't we still waiting for C# 6 to be officially released? Yes, we are. If you'll recall, the primary feature added to C# 5 was support for asynchronous operations through the async/await keywords. Most .NET programmers are only getting around to using this rather far- and deep-reaching feature, to say nothing of the new C# 6 features that are almost officially available. C# 6 brings the following features with it and can be used in the CTP versions of "Visual Studio 2015" or downloaded from "the Roslyn project" . Some of the more "interesting features of C# 6" are: * Auto-Property Initializers: initialize a property in the declaration rather than in the constructor or on an otherwise unnecessary local variable. * Out Parameter Declaration: An out parameter can now be declared inline with var or a specific type. This avoids the ugly variable declaration outside of a call to a Try* method. * Using Static Class: using can now be used with with a static class as well as a namespace. Direct access to methods and properties of a static class should clean up some code considerably. * String Interpolation: Instead of using string.Format() and numbered parameters for formatting, C# 6 allows expressions to be embedded directly in a string (á la PHP): e.g. "{Name} logged in at {Time}" * nameof(): This language feature gets the name of the element passed to it; useful for data-binding, logging or anything that refers to variables or properties. * Null-conditional operator: This feature reduces conditional, null-checking cruft by returning null when the target of a call is null. E.g. company.People?[0]?.ContactInfo?.BusinessAddress.Street includes three null-checks [Looking ahead to C# 7] If the idea of using await correctly or wrapping your head around the C# 6 features outlined above doesn't already make your poor head spin, then let's move on to language features that aren't even close to being implemented yet. That said, the "first set of design notes for C# 7" by Mads Torgerson include several interesting ideas as well. * Pattern-matching: C# has been ogling its similarly named colleague F# for a while. One of the major ideas on the table for C# is improving the ability to represent as well as match against various types of pure data, with an emphasis on immutable data. * Metaprogramming: Another focus for C# is reducing boilerplate and capturing common code-generation patterns. They're thinking of delegation of interfaces through composition. Also welcome would be an improvement in the expressiveness of generic constraints. Related User Voice issues: * "Expand Generic Constraints for constructors" * "[p]roper (generic) type ali[a]sing" * Controlling Nullability: Another idea is to be able to declare reference types that can never be null at compile-time (where reasonable -- they do acknowledge that they may end up with a "less ambitious approach"). * Readonly parameters and locals: Being able to express when change is allowed is a powerful form of expressiveness. C# 7 may include the ability to make local variables and parameters readonly. This will help avoid accidental side-effects. * Lambda capture lists: One of the issues with closures is that they currently just close over any referenced variables. The compiler just makes this happen and for the most part works as expected. When it doesn't work as expected, it creates subtle bugs that lead to leaks, race conditions and all sorts of hairy situations that are difficult to debug. If you throw in the increased use of and nesting of lambda calls, you end up with subtle bugs buried in frameworks and libraries that are nearly impossible to tease out. The idea of this feature is to allow a lambda to explicitly capture variables and perhaps even indicate whether the capture is read-only. Any additional capture would be flagged by the compiler or tools as an error.Contracts(!): And, finally, this is the feature I'm most excited about because I've been waiting for integrated language support for Design by Contract for literally decades [1], ever since I read the "Object-Oriented Software Construction 2" (OOSC2) for the first time. The design document doesn't say much about it, but mentions that ".NET already has a contract system", the weaknesses of which I've "written about before" . Torgersen writes: "When you think about how much code is currently occupied with arguments and result checking, this certainly seems like an attractive way to reduce code bloat and improve readability." ...and expressiveness and provability! There are a bunch of User Voice issues that I can't encourage you enough to vote for so we can finally get this feature: * "Integrate Code Contracts more deeply in the .NET Framework" * "Integrate Code Contract Keywords into the main .Net Languages" With some or all of these improvements, C# 7 would move much closer to a provable language at compile-time, an improvement over being a safe language at run-time. We can already indicate that instance data or properties are readonly. We can already mark methods as static to prevent the use of this. We can use ReSharper [NotNull] attributes to (kinda) enforce non-null references without using structs and incurring the debt of value-passing and -copying semantics. I'm already quite happy with C# 5, but if you throw in some or all of the stuff outlined above, I'll be even happier. I'll still have stuff I can think of to increase expressiveness -- covariant return types for polymorphic methods or anchored types or relaxed contravariant type-conformance -- but this next set of features being discussed sounds really, really good. -------------------------------------------------------------------------------- [1] I love the features of the language "Eiffel" , but haven't ever been able to use it for work. The tools and IDE are a bit stuck in the past (very dated on Windows; X11 required on OS X). The language is super-strong, with native support for contracts, anchored types, null-safe programming, contravariant type-conformance, covariant return types and probably much more that C# is slowly but surely including with each version. Unfair? I've been writing about this progress for years (from newest to oldest): * ".NET 4.5.1 and Visual Studio 2013 previews are available" * "A provably safe parallel language extension for C#" * "Waiting for C# 4.0: A casting problem in C# 3.5" * "Microsoft Code Contracts: Not with a Ten-foot Pole" * "Generics and Delegates in C#" * "Wildcard Generics" (this one was actually about Java) * "An analysis of C# language design" * "Static-typing for languages with covariant parameters" * "What is .NET?" ]]> https://www.earthli.com/news/view_article.php?id=3123 2015-03-07T08:11:14+01:00 https://earthli.com/users/marco

of these series, we discussed applications, which provide the model and data provider, and sessions, which encapsulate high-level data context. In "part II" , we covered command types and inputs to the data pipeline. In this article, we're going to take a look at the data pipeline itself. ... ]]>

of these series, we discussed applications, which provide the model and data provider, and sessions, which encapsulate high-level data context. In "part II" , we covered command types and inputs to the data pipeline. In this article, we're going to take a look at the data pipeline itself. 1. "Applications & Sessions" 2. "Command types & inputs" 3. The Data Pipeline 4. Builders & Commands 5. Contexts and Connections 6. Sessions, resources & objects [Overview] [image]The primary goal of the data pipeline is, of course, to correctly execute each query to retrieve data or command to store, delete or refresh data. The diagram to the right shows that the pipeline consists of several data handlers. Some of these refer to data sources, which can be anything: an SQL database or a remote service. [1] The name "pipeline" is only somewhat appropriate: A command can jump out anywhere in the pipeline rather than just at the opposite end. A given command will be processed through the various data handlers until one of them pronounces the command to be "complete". [Command context: recap] In the previous parts, we learned that the input to the pipeline is an IDataCommandContext. To briefly recap, this object has the following properties: * Session: Defines the context within which to execute the command * Handler: Implements an abstraction for reading/writing values and flags to the objects (e.g. SetValue(IMetaProperty)); more detail on this later * Objects: The sequence of objects on which to operate (e.g. for save commands) or to return (e.g. for load commands) * ExecutableQuery: The query to execute when loading or deleting objects * MetaClass: The metadata that describes the root object in this command; more detail on this later as well [Handlers] Where the pipeline metaphor holds up is that the command context will always start at the same end. The ordering of data handlers is intended to reduce the amount of work and time invested in processing a given command. [Analyzers] The first stage of processing is to quickly analyze the command to handle cases where there is nothing to do. For example, * The command is to save or delete, but the sequence of Objects is empty * The command is to save or reload, but none of the objects in the sequence of Objects has changed * The command is to load data but the query restricts to a null value in the primary key or a foreign key that references a non-nullable, unique key. It is useful to capture these checks in one or more analyzers for the following reasons, 1. All drivers share a common implementation for efficiency checks 2. Optimizations are applied independent of the data sources used 3. Driver code focuses on driver-specifics rather than general optimization [Caches] If the analyzer hasn't categorically handled the command and the command is to load data, the next step is to check caches. For the purposes of this article, there are two things that affect how long data is cached: 1. If the session is in a transacted state, then only immutable data, data that was loaded before the transaction began or data loaded within that transaction can be used. Data loaded/saved by other sessions -- possibly to global caches -- is not visible to a session in a transaction with an isolationLevel stricter than RepeatableRead. 2. The metadata associated with the objects can include configuration settings that control maximum caching lifetime as well as an access-timeout. The default settings are good for general use but can be tweaked for specific object types. Caches currently include the following standard handlers [2]: * The ValueListDataHandler returns immutable data. Since the data is immutable, it can be used independent of the transaction-state of the session in which the command is executed. * The SessionCacheDataHandler returns data that's already been loaded or saved in this session, to avoid a call to a possibly high-latency back-end. This data is safe to use within the session with transactions because the cache is rolled back when a transaction is rolled back. [Data sources] If the analyzer and cache haven't handled a command, then we're finally at a point where we can no longer avoid a call to a data source. Data sources can be internal or external. [Databases] The most common type is an external database: * PostgreSql 8.x and higher (PostgreSql 9.x for schema migration) * Sql Server 2008 and higher (w/schema migration) * Mongo (no schema; no migration) * SQlite (not yet released) [Remoting] Another standard data source is the Quino remote application server, which provides a classic interface- and method-based service layer as well as mapping nearly the full power of Quino's generalized querying capabilities to an application server. That is, an application can smoothly switch between a direct connection to a database to using the remoting driver to call into a service layer instead. The remoting driver supports both binary and JSON protocols. Further details are also beyond the scope of this article, but this driver has proven quite useful for scaling smaller client-heavy applications with a single database to thin clients talking to an application server. [Custom/Aspect-based] And finally, there is another way to easily include "mini" data drivers in an application. Any metaclass can include an IDataHandlerAspect that defines its own data driver as well as its capabilities. Most implementations use this technique to bind in immutable lists of data. But this technique has also been used to load/save data from/to external APIs, like REST services. We can take a look at some examples in more detail in another article. The mini data driver created for use with an aspect can relatively easily be converted to a full-fledged data handler. [Local evaluation] The last step in a command is what Quino calls "local evaluation". Essentially, if a command cannot be handled entirely within the rest of the data pipeline -- either entirely by an analyzer, one or more caches or the data source for that type of object -- then the local analyzer completes the command. What does this mean? Any orderings or restrictions in a query that cannot be mapped to the data source (e.g. a C# lambda is too complex to map to SQL) are evaluated on the client rather than the server. Therefore, any query that can be formulated in Quino can also be evaluated fully by the data pipeline -- the question is only of how much of it can be executed on the server, where it would (usually) be more efficient to do so. Please see the article series that starts with "Optimizing data access for high-latency networks" for specific examples. In this article, we've learned a bit about the ways in which Quino retrieves and stores data using the data pipeline. In the next part, we’ll cover the topic “Builders & Commands”. -------------------------------------------------------------------------------- [1] E.g. Quino uses a ProtoBuf-like protocol to communicate with its standard application server. [1] There is an open issue to "Introduce a global cache for immutable objects or objects used not in a transaction" . ]]> https://www.earthli.com/news/view_article.php?id=3111 2015-02-28T18:36:41+01:00 https://earthli.com/users/marco

, we discussed applications -- which provide the model and data provider -- and sessions -- which encapsulate high-level data context. In this article, we're going to take a look at the command types & inputs 1. "Applications & Sessions" 2. Command types & inputs [1] 3. The Data Pipeline 4. Builders & ... ]]>

, we discussed applications -- which provide the model and data provider -- and sessions -- which encapsulate high-level data context. In this article, we're going to take a look at the command types & inputs 1. "Applications & Sessions" 2. Command types & inputs [1] 3. The Data Pipeline 4. Builders & Commands 5. Contexts and Connections 6. Sessions, resources & objects [Overview] [image]Before we can discuss how the pipeline processes a given command, we should discuss what kinds of commands the data driver supports and what kind of inputs the caller can pass to it. As you can well imagine, the data driver can be used for CRUD -- to create, read, update and delete and also to refresh data. In the top-right corner of the diagram to the right, you can see that the only input to the pipeline is an IDataCommandContext. This object comprises the inputs provided by the caller as well as command-specific state used throughout the driver for the duration of the command. [Command types] A caller initiates a command with either a query or an object graph, depending on the type of command. The following commands and inputs are supported: * Load: returns a cursor for the objects that match a query * Count: returns the number of objects that match a query * Save: saves an object graph * Reload: refreshes the data in an object graph * Delete: deletes an object graph or the objects that match a query [Queries] A query includes information about the data to return (or delete). * Metadata: The meta-class represents the type of the root object for the command. For example, a "person" or "company". * Filtering: Filters restrict the objects to return. A filter can address properties of the root object, but also properties of objects related to the root object. A caller can query for people whose first names start with the letter "m" -- FirstName %~ 'm' [2] -- or the caller can find all people which belong to a company whose name starts with the letter "e" -- Company.FirstName %~ 'e'. The context for these expressions is naturally the meta-class mentioned above. Additionally, the metadata/model can also include default filters to include. * Ordering: Orderings that determine in which order the data is returned. Orderings are also specified with the expression language, but are usually simpler, like ordering first by LastName and then by FirstName. More complex expressions are supported -- for example, you could use the expression "{LastName}, {FirstName}", which sorts by a formatted string [3] -- but be aware that many data stores have limited support for complex expressions in orderings. Orderings are ignored in a query when used to delete objects. Queries are a pretty big topic and we've only really scratched the surface so far. Quino has its own query language -- QQL -- the specification for which weighs in at over 80 pages, but that's a topic for another day. [Object graphs] An object graph consists of a sequence of root objects and the sub-objects available along relations defined in the metadata. It's actually simpler than it perhaps sounds. Let's use the example above: a person is related to a single company, so the graph of a single person will include the company as well (if the object is loaded and/or assigned). Additionally, the company defines a relation that describes the list of people that belong to it. The person=>company relationship is complementary to the company=>person relationship. We call person=>company a 1-1 relation, while company=>person is a 1-n relation. The following code creates two new companies, assigns them to three people and saves everything at once. var encodo = new Company { Name = "Encodo Systems AG" }; var other = new Company { Name = "Not Encodo" }; var people = new [] { new Person { FirstName = "John", LastName = "Doe", Company = other }, new Person { FirstName = "Bob", LastName = "Smith", Company = encodo }, new Person { FirstName = "Ted", LastName = "Jones", Company = encodo } }; Session.Save(people); The variable people above is an object graph. The variables encodo and other are also object graphs, but only to parts of the first one. From people, a caller can look up people[0].Company, which is other. The graph contains cycles, so people[0].Company.People[0].Company is also other. From encodo, the caller can get to other people in the same company, but not to people in the other company, for example, encodo.People[0] gets "Bob Smith" and encodo.People[0].Company.People[1] gets "Ted Jones". As with queries, object graphs are a big topic and are strongly bound to the kind of metadata available in Quino. Another topic for another day. [Determining Inputs] Phew. We're almost to the point where we can create an IDataCommandContext to send into the data pipeline. * We have an IDataSession and "know why we need it" * We know what type of command we want to execute (e.g. "Load") * We have either a query or an object graph With those inputs, Quino has all it needs from the caller. A glance at the top-left corner of the diagram above shows us that Quino will determine an IMetaClass and an IMetaObjectHandler from these inputs and then use them to build the IDataCommandContext. An IQuery has a MetaClass property, so that's easy. With the meta-class and the requested type of object, the data driver checks a list of registered object-handlers and uses the first one that says it supports that type. If the input is an object graph, though, the object-handler is determined first and then the meta-class is obtained from the object-handler using a root object from the graph. Most objects will inherit from GenericObject which implements the IPersistable interface required by the standard object handler. However, an application is free to implement an object handler for other base classes -- or no base class at all, using reflection to get/set values on POCOs. That is, however, an exercise left up to the reader. At this point, we have all of our inputs and can create the IDataCommandContext. In the next part, we'll take a look at the "Data Pipeline" through which this command context travels. -------------------------------------------------------------------------------- [1] You'll notice, perhaps, that this topic is new to this article. I'm expanding the series as I go along, trying to provide enough information to understand the process while keeping the individual blog entries to a digestible size. [1] "%~" is actually the case-insensitive begins-with operator. You can find out more about comparison operators in the "Quino documentation" . Browse to "Encodo Base Library" and then "Expressions". [1] For more information on how to use Quino's unique take on interpolated strings, see the documentation in the footnote above. ]]> https://www.earthli.com/news/view_article.php?id=3109 2015-02-21T08:02:16+01:00 https://earthli.com/users/marco

(); people.Query.WhereEquals(Person.Fields.FirstName, "john"); session.Delete(people); session.Save(new Person { FirstName = "bob", LastName = "doe" }); } In this series, we're going to answer the following questions...and probably many more. * Where does the data come from? * What kind of sources are supported? How? * Is at least some of the data cached? * Can I influence the cache? * What is a session? Why do I need one? * Wait...what is the application? Let's tackle the last two questions first. [Application] The application defines common configuration information. The most important bits for the ORM are as follows: * Model: The model is the central part of any Quino application. The model defines entities, their properties, relationships between entities and so on. Looking at the example above, the model will include a definition for a Person, which has at least the two properties LastName and FirstName. There is probably an entity named Company as well, with a one-to-many relationship to Person. As you can imagine, Quino uses this information to formulate requests to data stores that contain data in this format. [1] For drivers that support it, Quino also uses this information in order to create that underlying data schema. [2] * DataProvider: The data provider encapsulates all of the logic and components needed to map the model to data sources. This is the part of the process on which this series will concentrate. * ConfigurationData: The configuration data describes which parts of the model are connected to which parts of the data provider. The default is, of course, that the entire model is mapped to a single data source. However, even in that case, the configuration indicates which data source: Sql Server? PostgreSql? A remote application server (2nd tier)? With a high-level API as described above, all of these decisions can be made in the configuration rather than assumed throughout the application. Yes, this means that you can change your Quino application from a two-tier to a three-tier application with a single configuration change. [Sessions] So that's the application. There is a single shared application for a process. But in any non-trivial application -- and any non-desktop application -- we will have multiple data requests running, possibly in different threads of execution. * Each request in a web application is a separate data context. Changes made in one request should not affect any other request. Each request may be authenticated as a different user. * A remote application-server is very similar to a web application. It handles requests from multiple users. Since it's generally the second layer, it will most likely have direct connections to one or more databases. In this case, it will probably be in charge of executing business logic, most likely in a database transaction. In that case, we definitely don't want one request using the transaction context from another request. * Even a non-web client-side application may want to execute some logic in the background or in a separate thread. In those cases, we probably want to keep the data used there separate from the data or objects used to render the other parts of the application. That's where sessions come in. The session encapsulates a data context, which contains the following information: * Application: The application will, as described above, tell the session which model and data provider to use. * Current user: For those familiar with ASP.NET, this is very similar to the HttpContext.Current.User but generalized to be available in any Quino application. All data requests over a session are made in the context of this user. * Access control: The access control provides information about the security context of an application. An application generally uses the access control to perform authorization checks. * Cache: Each session also has its own cache. There are global caches, but those are for immutable data. The session's cache is always available, even when using transactions. * ConnectionManager: Many external data sources have transactable/shared state in the form of a connection. As with data, connections can sometimes be shared between sessions and sometimes they can't. The connection manager takes care of knowing all of that for you. If we go back to the original code sample, we now know that creating a new session with CreateSession() creates a new data context, with its own user and its own data cache. Since we didn't pass in any credentials, the session uses the default credentials for the application. [3] All data access made on that session is nicely shielded and protected from any data access made in other sessions (where necessary, of course). So now we're no closer to knowing how Quino works with data on our behalf, but we've taken the first step: we know all about one of the main inputs to the data driver, the session. In the next part, we'll cover the topic "The Data Pipeline". -------------------------------------------------------------------------------- [1] The domain model is used for everything in a Quino application -- not just the ORM and for schema-migration. We use the model to generate C# code like concrete ORM objects, metadata references (e.g. the Person.Fields.FirstName in the example), or view models, DTOs or even client-side TypeScript definitions. We also use the model to generate user interfaces -- both for entire desktop-application interfaces but also for HTML helpers to build MVC views. [1] See the article "Schema migration in Quino 1.13" for more information on how that works. [1] This is code that you might use in a single-user application. In a server application, you would most likely just use the session that was created for your request by Quino. If an application wants to create a new session, but using the same user as an existing session, it would call: var requestCredentials = requestSession.AccessControl.CurrentUser.CreateCredentials(); using (var session = application.CreateSession(requestCredentials)) { // Work with session } ]]> https://www.earthli.com/news/view_article.php?id=3107 2015-02-11T07:11:51+01:00 https://earthli.com/users/marco

https://www.earthli.com/news/view_article.php?id=3064 2014-11-16T00:20:42+01:00 https://earthli.com/users/marco

, I explained how we were using NDepend to clean up dependencies and the architecture of our Quino framework. You have to start somewhere, so I started with the two base assemblies: Quino and Encodo. Encodo only has dependencies on standard .NET assemblies, so let's start... ]]>

, I explained how we were using NDepend to clean up dependencies and the architecture of our Quino framework. You have to start somewhere, so I started with the two base assemblies: Quino and Encodo. Encodo only has dependencies on standard .NET assemblies, so let's start with that one. The first step in cleaning up the Encodo assembly is to remove dependencies on the Tools namespace. There seems to be some confusion as to what belongs in the Core namespace versus what belongs in the Tools namespace. There are too many low-level classes and helpers in the Tools namespace. Just as a few examples, I moved the following classes from Tools to Core: * BitTools * ByteTools * StringTools * EnumerableTools The names kind of speak for themselves: these classes clearly belong in a core component and not in a general collection of tools. Now, how did I decide which elements to move to core? NDepend helped me visualize which classes are interdependent. [Direct Dependencies] [image]We see that EnumerableTools depends on StringTools. I'd just moved EnumerableTools to Encodo.Core to reduce dependence on Encodo.Tools. However, since StringTools is still in the Tools namespace, the dependency remains. This is how examining dependencies really helps clarify a design: it's now totally obvious that something as low-level as StringTools belongs in the Encodo.Core namespace and not in the Encodo.Tools namespace, which has everything but the kitchen sink in it. [image]Another example in the same vein is shown to the left, where we examine the dependencies of MessageTools on Encodo.Tools. The diagram explains that the colors correspond to the two dependency directions. [1] We would like the Encodo.Messages namespace to be independent of the Encodo.Tools namespace, so we have to consider either (A) removing the references to ExceptionTools and OperatingSystemTools from MessageTools or (B) moving those two dependencies to the Encodo.Core namespace. Choice (A) is unlikely while choice (B) beckons with the same logic as the example above: it's now obvious that tools like ExceptionTools and OperatingSystemTools belong in Encodo.Core rather than the kitchen-sink namespace. [Indirect Dependencies] Once you're done cleaning up your direct dependencies, you still can't just sit back on your laurels. Now, you're ready to get started looking at indirect dependencies. These are dependencies that involve more than just two namespaces that use each other directly. NDepend displays these as red bounding blocks. The documentation indicates that these are probably good component boundaries, assuming that the dependencies are architecturally valid. NDepend can only show you information about your code but can't actually make the decisions for you. As we saw above, if you have what appear to be strange or unwanted dependencies, you have to decide how to fix them. In the cases above, it was obvious that certain code was just in the wrong namespace. In other cases, it may simply be a few bits of code are defined at too low a level. [Improper use of namespaces] For example, our standard practice for components is to put high-level concepts for the component at the Encodo. namespace. Then we would use those elements from sub-namespaces, like Encodo..Utils. However, we also ended up placing types that then used that sub-namespace in the upper-level namespace, like ComponentNameTools.SetUpEnvironment() or something like that. The call to SetUpEnvironment() references the Utils namespace which, in turn, references the root namespace. This is a direct dependency, but if another namespace comes between, we have an indirect dependency. This happens quite quickly for larger components, like Encodo.Security. The screenshots below show a high-level snapshot of the indirect dependencies in the Encodo assembly and then also a detail view, with all sub-namespaces expanded. The detail view is much larger but shows you much more information about the exact nature of the cycle. When you select a red bounding box, another panel shows the full details and exact nature of the dependency. [image][image][image] [Base Camp Two: base library almost cleaned up] [image][image]After a bunch of work, I've managed to reduce the dependencies to a set of interfaces that are clearly far too dependent on many subsystems. * ICoreConfiguration: references configuration options for optional subsystems like the software updater, the login, the incident reporter and more * ICoreFeedback: references feedbacks for several optional processes, like software-update, logins and more * ICoreApplication: references both the core configuration and feedback The white books for NDepend claim that "[t]echnically speaking, the task of merging the source code of several assemblies into one is a relatively light one that takes just a few hours." However, this assumes that the code has already been properly separated into non-interdependent namespaces that correspond to components. These components can then relatively easily be extracted to separate assemblies. The issue that I have above with the Encodo assembly is a thornier one: the interfaces themselves embody a pattern that is inherently non-decoupling. I need to change how the configuration and feedback work completely in order to decouple this code. [Roadmap for startup and configuration] To that end, I've created an issue in the issue-tracker for Quino, "QNO-4659" [2], titled "Re-examine how the configuration, feedback and application work together". The design of these components predates our introduction of a service locator, which means it's much more tightly coupled (as you can see above). After some internal discussion, we've decided to change the design of the Encodo and Quino library support for application-level configuration and state. Merge the configuration and application To date, the configuration has contained all of the information necessary to run an application. The configuration was more-or-less stateless and corresponded to the definition of an application, akin to how a class is the underlying stateless definition, while an object is an instance of that definition. In practice, though, we always use a single application per configuration and the distinction is irrelevant, for all practical purposes. This will simplify all referencing code, as we will no longer need to pass around an IApplication. Move the feedback to the service locator Instead of treating the feedback like a first-class citizen, with a direct reference on the application, make consumers use the service locator to retrieve an instance. This will remove the remaining generic argument in the definition of IApplication, leaving us with a base interface that is free of generic arguments. Move specific configuration objects to the service locator The specific sub-interfaces that introduce dependencies are as follows: * IncidentReporter * SoftwareUpdater * CommandSetManager * LocationManager * ConnectionSettingsManager Any components that currently reference the properties on the ICoreConfiguration can use the service locator to retrieve an instance instead. Move specific settings to sub-objects The configuration object is not only dependent on sub-objects, but is also overloaded with individual settings that are only used by very few specific sub-components. These will also be extracted into interfaces and moved into the service locator. * ILoginConfiguration * ISoftwareUpdateConfiguration * IFileLogConfiguration As you can see, while NDepend is indispensable for finding dependencies, it can -- along with a good refactoring tool (we use ReSharper) -- really only help you clean up the low-hanging fruit. While I started out trying to split assemblies, I've now been side-tracked into cleaning up an older and less--well-designed component -- and that's a very good thing. There are some gnarly knots that will feel nearly unsolvable -- but with a good amount of planning, those can be re-designed as well. As I mentioned in the previous article, though, we can do so only because we're making a clean break from the 1.x version of Quino instead of trying to maintain backward compatibility. It's worth it, though: the new design already looks much cleaner and is much more easily explained to new developers. Once that rewrite is finished, the Encodo assembly should be clean and I'll use NDepend to find good places to split up that rather large assembly into sensible sub-assemblies. -------------------------------------------------------------------------------- [1] There is a setting to turn off showing the green dependencies -- where the row depends on the column -- to make it easier to read the matrix. If you do that, though, you have to make sure to select the class from which you're trying to remove dependencies in the column. For example, if class A and B are interdependent, but A should not rely on B, you should make sure A is showing in the column. You can then examine dependencies on row B -- and then remove them. This works very nicely with both direct and indirect dependencies. [1] This link is to the Quino issue tracker, which requires a login. ]]> https://www.earthli.com/news/view_article.php?id=3058 2014-11-12T22:23:25+01:00 https://earthli.com/users/marco

to analyze code dependencies. The trial license is fourteen days; needless to say, I got only one afternoon in before I was distracted by other duties. That was enough, however, to convince me that it was worth the... ]]>

to analyze code dependencies. The trial license is fourteen days; needless to say, I got only one afternoon in before I was distracted by other duties. That was enough, however, to convince me that it was worth the $375 to continue to clean up Quino with NDepend. I decided to wait until I had more time before opening my wallet. In the meantime, however, Patrick Smacchia of NDepend approached me with a free license if I would write about my experiences using NDepend on Encodo's blog. I'm happy to write about how I used the tool and what I think it does and doesn't do. [1] [History & Background] [image]We started working on Quino in the fall of 2007. As you can see from the first commit, the library was super-small and comprised a single assembly. Fast-forward seven years and Version 1.13 of Quino has 66 projects/assemblies. That's a lot of code and it was long past time to take a look a more structured look at how we'd managed the architecture over the years. I'd already opened a branch in our Quino repository called feature/dependencyChanges and checked in some changes at the beginning of July. Those changes had come as a result of the first time I used NDepend to find a bunch of code that was in the wrong namespace or the wrong assembly, architecturally speaking. [Sidebar: Keeping branches mergeable] I wasn't able to continue using this branch, though, for the following reasons. 1. I got the hang of NDepend relatively quickly and got a bit carried away. Using ReSharper, I was able to make a lot of changes and fixes in a relatively short amount of time. 2. I checked in all of these changes in one giant commit. 3. I did this all five months ago. 4. There have been hundreds of subsequent commits on the master branch, many of which also include global refactoring and cleanup. 5. As a result of the above, merging master into feature/dependencyChanges is more trouble than it's worth. [Release Methodology] With each Quino change and release, we try our hardest to balance backward-compatibility with maintainability and effort. If it's easy enough to keep old functionality under an old name or interface, we do so. We mark members and types obsolete so that users are given a warning in the compiler but can continue using the old code until they have time to upgrade. These obsolete members are removed in the next major or minor upgrade. Developers who have not removed their references to obsolete members will at this point be greeted with compiler errors. In all cases, the user can find out from Quino's release notes how they should fix a warning or error. The type of high-level changes that we have planned necessitate that we make a major version-upgrade, to Quino 2.0. In this version, we have decided not to maintain backward-compatibility in the code with Obsolete attributes. However, where we do make a breaking change -- either by moving code to new or different assemblies or by changing namespaces -- we want to maintain a usable change-log for customers who make the upgrade. The giant commit that I'd made previously was not a good start. [Take Two] Since some of these changes will be quite drastic departures in structure, we want to come up with a plan to make merging from the master branch to the feature/dependencyChanges branch safer, quicker and all-around easier. I want to include many of the changes I started in the feature/dependencyChanges branch, but would like to re-apply those changes in the following manner: * Split the giant commit into several individual commits, each of which encapsulates exactly one change; smaller commits are much easier to merge * Document breaking changes in the release notes for Quino 2.0 * Blog about/document the process of using NDepend to clean up Quino [2] So, now that I'm ready to start cleaning up Quino for version 2.0, I'll re-apply the changes from the giant commit, but in smaller commits. At the same time, I'll use NDepend to find the architectural breaks that caused me to make those changes in the first place and document a bit of that process. [Setting up the NDepend Project] I created an NDepend project and attached it to my solution. Version 1.13 of Quino has 66 projects/assemblies, of which I chose the following "core" assemblies to analyze. [image] I can change this list at any time. There are a few ways to add assemblies. Unfortunately, the option to "Add Assemblies from VS Solution(s)" showed only 28 of the 66 projects in the Quino solution. I was unable to determine the logic that led to the other 38 projects not being shown. When I did select the projects I wanted from the list, the assemblies were loaded from unexpected directories. For example, it added a bunch of core assemblies (e.g. Encodo.Imaging) from the src/tools/Quino.CodeGenerator/bin/ folder rather than the src/libraries/Encodo.Imaging/bin folder. I ended up just taking the references I was offered by NDepend and added references to Encodo and Quino, which it had not offered to add. [3] [The NDepend Dashboard] Let's take a look at the initial NDepend Dashboard. [image] There's a lot of detail here. The initial impression of NDepend can be a bit overwhelming, I supposed, but you have to remember the sheer amount of interdependent data that it shows. As you can see on the dashboard, not only are there a ton of metrics, but those metrics are also tracked on a time-axis. I only have one measurement so far. Any assemblies not included in the NDepend project are considered to be "third-party" assemblies, so you can see external dependencies differently than internal ones. There is also support for importing test-coverage data, but I haven't tried that yet. There are a ton of measurements in there, some of which interest me and others that don't, or with which I disagree. For example, over 1400 warnings are in the Quino* assemblies because the base namespace -- Encodo.Quino -- doesn't correspond to a file-system folder -- it expects Encodo/Quino, but we use just Quino. Another 200 warnings are to "Avoid public methods not publicly visible", which generally means that we've declared public methods on internal, protected or private classes. The blog post "Internal or public?" by Eric Lippert covered this adequately and came to the same conclusion that we have: you actually should make methods public if they are public within their scope. There are some "White Books" about namespace and assembly dependencies that are worth reading if you're going to get serious about dependencies. There's a tip in there about turning off "Copy Local" on referenced assemblies to drastically increase compilation speed that we're going to look into. [Dependencies and cycles] One of the white books explains how to use namespaces for components and how to "levelize" an architecture. This means that the dependency graph is acyclic -- that there are no dependency cycles and that there are certainly no direct interdependencies. The initial graphs from the Encodo and Quino libraries show that we have our work cut out for us. [image][image][image] The first matrix shows the high-level view of dependencies in the Encodo and Quino namespaces. Click the second and third to see some initial dependency issues within the Encodo and Quino assemblies. That's as far as I've gotten so far. Tune in next time for a look at how we managed to fix some of these dependency issues and how we use NDepend to track improvement over time. -------------------------------------------------------------------------------- [1] I believe that takes care of full disclosure. [1] This is something I'd neglected to do before. Documenting this process will help me set up a development process where we use NDepend more regularly -- more than every seven years -- and don't have to clean up so much code at once. [1] After having read the recommendations in the NDepend White Book -- "Partitioning code base through .NET assemblies and Visual Studio projects" (PDF) -- it's clear why this happens: NDepend recommends using a single /bin folder for all projects in a solution. ]]> https://www.earthli.com/news/view_article.php?id=3057 2014-11-12T22:14:18+01:00 https://earthli.com/users/marco

provides a fascinating and in-depth look at how a modern execution engine optimizes code for a highly dynamic language like JavaScript. To make a long story short: the compiler(s) and execution engine optimize by profiling and... ]]>

provides a fascinating and in-depth look at how a modern execution engine optimizes code for a highly dynamic language like JavaScript. To make a long story short: the compiler(s) and execution engine optimize by profiling and analyzing code and lowering it to runtimes of ever decreasing abstraction to run as the least dynamic version possible. [A brief history lesson] What does it mean to "lower" code? A programming language has a given level of abstraction and expressiveness. Generally, the more expressive it is, the more abstracted it is from code that can actually be run in hardware. A compiler transforms or translates from one language to another. When people started programming machines, they used punch cards. Punch cards did not require any compilation because the programmer was directly speaking the language that the computer understood. The first layer of abstraction that most of us -- older programmers -- encountered was assembly language, or assembler. Assembly code still has a more-or-less one-to-one correspondence between instructions and machine-language codes but there is a bit of abstraction in that there are identifiers and op-codes that are more human-readable. Procedural languages introduced more types of statements like loops and conditions. At the same time, the syntax was abstracted further from assembler and machine code to make it easier to express more complex concepts in a more understandable manner. At this point, the assembler (which assembled instructions into machine op-codes) became a compile which "compiled" a set of instructions from the more abstract language. A compiler made decisions about how to translate these concepts, and could make optimization decisions based on registers, volatility and other settings. In time, we'd graduated to functional, statically typed and/or object-oriented languages, with much higher levels of abstraction and much more sophisticated compilers. Generally, a compiler still used assembly language as an intermediate format, which some may remember from their days working with C++ or Pascal compilers and debuggers. In fact, .NET languages are also compiled to IL -- the "Intermediate Language" -- which corresponds to the instruction set that the .NET runtime exposes. The runtime compiles IL to the underlying machine code for its processor, usually in a process called JIT -- Just-In-Time compilation. That is, in .NET, you start with C#, for example, which the compiler transforms to IL, which is, in turn, transformed to assembler and then machine code by the .NET runtime. [Static vs. Dynamic compilation] A compiler and execution engine for a statically typed language can make assumptions about the types of variables. The set of possible types is known in advance and types can be checked very quickly in cases where it's even necessary. That is, the statically typed nature of the language allows the compiler to reason about a given program without making assumptions. Certain features of a program can be proven to be true. A runtime for a statically typed language can often avoid type checks entirely. It benefits from a significant performance boost without sacrificing any runtime safety. The main characteristic of a dynamic language like JavaScript is that variables do not have a fixed type. Generated code must be ready for any eventuality and must be capable of highly dynamic dispatch. The generated code is highly virtualized. Such a runtime will execute much more slowly than a comparable statically compiled program. [Profile-driven compilation] Enter the profile-driven compiler, introduced in WebKit. From the article, "The only a priori assumption about web content that our engine makes is that past execution frequency of individual functions is a good predictor for those functions’ future execution frequency." Here a "function" corresponds to a particular overload of a set of instructions called with parameters with a specific set of types. That is, suppose a JavaScript function is declared with one parameter and is called once with a string and 100 times with an integer. WebKit considers this to be two function overloads and will (possibly) elect to optimize the second one because it is called much more frequently. The first overload will still handle all possible types, including strings. In this way, all possible code paths are still possible, but the most heavily used paths are more highly optimized. "All of the performance is from the DFG’s type inference and LLVM’s low-level optimizing power. [...] "Profile-driven compilation implies that we might invoke an optimizing compiler while the function is running and we may want to transfer the function’s execution into optimized code in the middle of a loop; to our knowledge the FTL is the first compiler to do on-stack-replacement for hot-loop transfer into LLVM-compiled code." Depending on the level of optimization, the code contains the following broad sections: * Original: code that corresponds to instructions written by the author * Profiling: code to analyze which types actually appear in a given code path * Switching: code to determine when a function has been executed often enough to warrant further optimization * Bailout code to abandon an optimization level if any of the assumptions made at that level no longer apply [image] While WebKit has included some form of profile-driven compilation for quite some time, the upcoming version is the first to carry the same optimization to LLVM-generated machine code. I recommend reading the "whole article" if you're interested in more detail, such as how they avoided LLVM compiler performance issues and how they integrated this all with the garbage collector. It's really amazing how much that we take for granted the WebKit JS runtime treats as "hot-swappable". The article is quite well-written and includes diagrams of the process and underlying systems. ]]> https://www.earthli.com/news/view_article.php?id=3052 2014-10-31T10:39:12+01:00 https://earthli.com/users/marco

. The "full list of issues" is also available for those with access to the Encodo issue tracker. [Highlights] [Data & Schema] * Applications can now choose a base class for generated ORM objects. ("QNO-3107" ) * Data ... ]]>

. The "full list of issues" is also available for those with access to the Encodo issue tracker. [Highlights] [Data & Schema] * Applications can now choose a base class for generated ORM objects. ("QNO-3107" ) * Data driver: made various bug fixes and improvements. ("QNO-4538" , "QNO-4554" , "QNO-4551" ) * Improved schema migration for fields and constraint-violation messages under SQL Server. ("QNO-4490" , "QNO-4111" , "QNO-4582" ) * Improved the console migrator and APIs, input queries and exit-code handling for console applications in general. Also changed the default command from [R]efresh to [S]how differences. ("QNO-4649" , "QNO-4646" , "QNO-4648" , "QNO-4650" , "QNO-4651" , "QNO-4615" , "QNO-4645" , "QNO-4616" ) [Remoting & services] * Fixed several issues in the remoting driver (client and server parts). ("QNO-4626" , "QNO-4630" , "QNO-4631" , "QNO-4388" , "QNO-4575" , "QNO-4629" , "QNO-4573" , "QNO-4625" , "QNO-4633" , "QNO-4575" ) * Added a runner for Windows services that allows debugging and shows logging output for applications that use the CoreServiceBase, which extends the standard .NET ServiceBase. The runner is available in the Encodo.Service assembly. [Web] * Improved default and custom authentication in web applications and the remoting server. Also improved support for authorization for remote-method routes as well as MVC controllers. * Improved configuration, error-handling and stability of the HttpApplicationBase, especially in situations where the application fails to start. Error-page handling was also improved, including handling for Windows Event Log errors. * Improved appearance of the web-based schema migrator. ("QNO-4559" , "QNO-4561" , "QNO-4563" , "QNO-4548" , "QNO-4487" , "QNO-4486" , "QNO-4488" ) [Winform] * Data-provider statistics: improved the WinForm-based statistics form. ("QNO-4231" , "QNO-4545" , "QNO-4546" ) * Standard forms: updated the standard WinForm about window and splash screen to use Encodo web-site CI. ("QNO-4529" ) [System & Tools] * Removed the dependency on the SmartWeakEvents library from Quino. ("QNO-4645" ); the Quino and Encodo assemblies now no longer have any external dependencies. * Image handling: the Encodo and Quino libraries now use the Windows Imaging Components instead of System.Drawing. ("QNO-4536" ) * Window 8.1: fixed culture-handling for en-US and de-CH that is broken in Windows 8.1. ("QNO-4534" , "QNO-4553" ) * R# annotations have been added to the Encodo assembly. Tell R# to look in the Encodo.Core namespace to use annotations like NotNull and CanBeNull with parameters and results. ("QNO-4508" ) * Generated code now includes a property that returns a ValueListObject for each enum property in the metadata. For example, for a property named State of type CoreState, the generated code includes the former properties for the enum and the foreign key backing it, but now also includes the ValueListObject property. This new property provides easy access to the captions. public CoreState State { ... } public ValueListObject StateObject { ... } public int? CoreStateIdId { ... }Improved the nant fix command in the default build tools to fix the assembly name as well. The build tools are available in bin/tools/build. See the src/demo/Demo.build file for an example on how to use the Nant build scripts for your own solutions. To change the company name used by the "fix" command, for example, add the following task override: * Fixed the implementation of IntegrateRemotableMethods to avoid a race condition with remote methods. Also improved the stability of the DataProvider statistics. ("QNO-4599" ) [Breaking changes] * The generic argument TRight has been removed from all classes and interfaces in the Encodo.Security.* namespace. In order to fix this code, just remove the int generic parameter wherever it was used. For example, where before you used the interface IUser, you should now use IUser ("QNO-4576" ). * The overridable method MetaAccessControl.DoGetAccessChecker() has been renamed to MetaAccessControl.GetAccessChecker(). * Renamed the Encodo.ServiceLocator.SimpleInjector.dll to Encodo.Services.SimpleInjector.dll and Quino.ServiceLocator.SimpleInjector.dll to Quino.Services.SimpleInjector.dll Also changed the namespace Quino.ServiceLocator to Encodo.Quino.Services. * Renamed HttpApplicationBase.StartMetaApplication() to CreateAndStartUpApplication(). * Classes may no longer contain properties with names that conflict with properties of IMetaReadable (e.g. Deleted, Persisted). The model will no longer validate until the properties have been renamed and the code regenerated. ("QNO-4185" ) * Removed StandardIntRights with integer constants and replaced it with StandardRights with string constants. * The IAccessControl.Check() and other related methods now accept a sequence of string rights rather than integers. * IMetaConfiguration.ConfigureSession() has been deprecated. The method will still be called but may have undesired side-effects, depending on why it was overridden. The common use was to initialize a custom AccessControl for the session. Continuing to do so may overwrite the current user set by the default Winform startup. Instead, applications should use the IDataSessionAccessControlFactory and IDataSessionFactory to customize the data sessions and access controls returned for an application. In order to attach an access control, take care to only set your custom access control for sessions that correspond to your application model. [1] internal class JobVortexDataSessionAccessControlFactory : DataSessionAccessControlFactory { public override IAccessControl CreateAccessControl(IDataSession session) { if (session.Application.Model.MetaId == JobVortexModelGenerator.ModelGuid) { return new JobVortexAccessControl(session); } return base.CreateAccessControl(session); } } The default length of the UserModule.User.PasswordHash property has been increased from 100 characters to 1000. This default is more sensible for implementations that use much longer validations tokens instead of passwords. To avoid the schema migration, revert the change by setting the property default length back to 0 in your application model, after importing the security module, as shown below. var securityModule = Builder.Include(); securityModule.Elements.Classes.User.Properties[ Encodo.Quino.Models.Security.Classes.SecurityUser.Fields.PasswordHash ].MaximumSize = 100;Application.Credentials has been removed. To fix references, retrieve the IUserCredentialsManager from the service locator. For example, the following code returns the current user: Session.Application.Configuration.ServiceLocator.GetInstance().Current If your application uses the WinformMetaConfigurationTools.IntegrateWinformPackages() or WinformDxMetaConfigurationTools.IntegrateWinformDxPackages(), then the IDataSession.AccessControl.CurrentUser will continue to be set correctly. If not, add the SingleUserApplicationConfigurationPackage to your application's configuration. The user in the remoting server will be set up correctly. Add the WebApplicationConfigurationPackage to web applications in order to ensure that the current user is set up correctly for each request. ("QNO-4596" ) * IDataSession.SyncRoot has been removed as it was no longer needed or used in Quino itself. Sessions should not be used in multiple threads, so there is no need for a SyncRoot. Code that uses it should be reworked to use a separate session for each thread. * Moved IMetaApplication.CreateSession() to an extension method. Add Encodo.Quino.App to the using clauses to fix any compile errors. * Removed IMetaApplication.DataProvider; use IMetaApplication.Configuration.DataProvider instead. ("QNO-4604" ) * The schema migration API has been completely overhauled. ISchemaChange and descendents has been completely removed. ISchemaAction is no longer part of the external API, although it is still used internally. The ISchemaChangeFactory has been renamed to ISchemaCommandFactory and, instead of creating change objects, which are then applied directly, returns ISchemaCommand objects, which can be either executed or transformed in some other way. IMigrateToolkit.GetActionFor() has also been replace with CreateCommands(), which mirrors the rest of the API by returning a sequence of commands to address a given ISchemaDifference. This release still has some commands that cannot be transformed to pure SQL, but the goal is to be able to generate pure SQL for a schema migration. ("QNO-993" , "QNO-4579" , "QNO-4581" , "4588" , "4591" , "QNO-4594" ) * IMigrateSchemaAspect.Apply() has been removed. All aspects will have to be updated to implement GetCommands() instead, or to use one of the available base classes, like UpdateDataAspectBase or ConvertPropertyTypeSchemaAspect. The following example shows how to use the UpdateDataAspectBase to customize migration for a renamed property. internal class ArchivedMigrationAspect : UpdateDataAspectBase { public ArchivedMigrationAspect() : base("ArchivedMigrationAspect", DifferenceType.RenamedProperty, ChangePhase.Instead) { } protected override void UpdateData(IMigrateContext context, ISchemaDifference difference) { using (var session = context.CreateSession(difference)) { session.ChangeAndSaveAll(UpdateArchivedFlag); } } private void UpdateArchivedFlag(Project obj) { obj.Archived = !obj.Archived; } } The base aspects should cover most needs; if your functionality is completely customized, you can easily pass your previous implementation of Apply() to a DelegateSchemaCommand and return that from your implementation of GetCommands(). See the implementation of UpdateDataAspectBase for more examples. ("QNO-4580" ) * MetaObjectIdEqualityComparer can no longer be constructed directly. Instead, use MetaObjectIdEqualityComparer.Default. * Renamed MetaClipboardControlDx.UpdateColorSkinaware() to MetaClipboardControlDx.UpdateSkinAwareColors(). * IMetaUnique.LogicalParent has been moved to IMetaBase. Since IMetaUnique inherits from IMetaBase, it is unlikely that code is affected (unless reflection or some other direct means was used to reference the property). ("QNO-4586" ) * IUntypedMessage has been removed; the AssociatedObject formerly found there has been moved to IMessage. * ITypedMessage.AssociatedObject has been renamed to ITypedMessage.TypedAssociatedObject. ("QNO-4647" ) * Renamed MetaObjectTools to MetaReadableTools. * Redefined the protected methods GenericObject.GetAsGuid() and GenericObject.GetAsGuidDefault as extension methods in MetaWritableTools. * IMetaFeedback.CreateGlobalContext() has been removed. Instead the IGlobalContext is created using the service locator. -------------------------------------------------------------------------------- [1] The schema migration creates a metadata model for your model -- meta-metadata -- and uses the Quino ORM to load data when importing a model from a database. If you aren't careful, as shown in the code example, then you'll attach your custom access control to the sessions created for the schema migration's data-access, which will more than likely fail when it tries to load user data from a table that does not exist in that model. ]]> https://www.earthli.com/news/view_article.php?id=3049 2014-10-24T12:26:25+02:00 https://earthli.com/users/marco

MetaClasses * Fields/Columns => MetaProperties * Indexes => MetaIndexes * Foreign Keys => MetaPaths If there is no further information in the database, then the mapper will have to use the raw model only. If, however, the database was created or is being maintained by Quino, then there is additional information stored in the metadata table mentioned above. The importer enhanced the raw model with this information, in order to improve mapping and difference-recognition. The metadata table contains all of the Quino modeling information that is not reflected in a standard database schema (e.g. the aforementioned MetaId). The data available in this table is currently: * SchemaIdentifier: the identifier used in the raw model/database schema * Identifier: the actual identifier of the metadata element that corresponds to the element identified by the SchemaIdentifier * MetaId: the unique id for the metadata element * ObjectType: the type of metadata (one of: class, property, index, path, model) * ParentMetaId: the unique id of the metadata element that is the logical parent of this one; only allowed to be empty for elements with ObjectType equal to "model" * Data: Custom data associated with the element, as key/value pairs * DataVersion: Identifies the format type of the "Data" element (1.0.0.0 corresponds to CSV) For each schema element in the raw model, the importer does the following: 1. Looks up the data associated with that SchemaIdentifier and ObjectType (e.g. "punchclock__person" and "class") 2. Updates the "Identifier" 3. Sets the "MetaId" 4. Loads the key/value pairs from the Data field and applies that data to the element [Generating a mapping] At this point, the imported model is ready and we can create a mapping between it and the application model. The imported model is called the source model while the application model is called the target model because we're migrating the "source" to match the "target". We generate a mapping by iterating the target model: 1. Find the corresponding schema element in the source model using MetaIds [1] 2. If an element can be found, create a mapping for those two elements 3. If no element can be found, create a mapping with the target element. This will cause the element to be created in the database. 4. For all elements in the source model that have no corresponding element in the target model, create a mapping with only the source element. This will cause the element to be dropped from the database. [Creating a migration plan] The important decisions have already been made in the mapping phase. At this point, the migrator just generates a migration plan, which is a list of differences that must be addressed in order to update the database to match the target model. * If the mapping has a source and target element * Create a difference if the element has been renamed * Create a difference if the element has been altered (e.g. a property has a different type or is now nullable; an index has new properties or is no longer unique; etc.) If the mapping has only a source, generate a difference that the element is unneeded and should be dropped. * If the mapping has only a target, generate a difference that the element is missing and should be created. This is the plan that is shown to the user by the various migration tools available with Quino. [2] [The advanced console-migrator commands] At this point, we can now understand what the advanced console-migrator commands mean. Significant mappings are those mappings which correspond to a difference in the database (create, drop, rename or alter). * Show significant mappings: show significant mappings to see more detail about the names on each side * Show significant mappings with unique ids: same as above, but also include the MetaIds for each side. Use this to debug when you suspect that you might have copy/pasted a MetaId incorrectly or inadvertently moved one. * Show all mappings: Same detail level as the first option, but with all mappings, including those that are 100% matches * Show all mappings with unique ids: same as above, but with MetaIds As already stated, the advanced options are really there to help a developer see why the migrator might be suggesting a change that doesn't correspond to expectations. [Generating commands for the plan] At this point, the migrator displays the list of differences that will be addressed by the migrator if the user chooses to proceed. What happens when the user proceeds? The migrator generates database-specific commands that, when executed against the database, will modify the schema of the database. [3] Commands are executed for different phases of the migration process. The phases are occasionally extended but currently comprise the following. * Initialize: perform any required initialization before doing anything to the schema * DropConstraintsAndIndexes: drop all affected constraints and indexes that would otherwise prevent the desired modification of the elements involved in the migration. * AddUpdateOrRenameSchema: Create new tables, columns and indexes and perform any necessary renaming. The changes in this phase are non-destructive * UpdateData: Perform any necessary data updates before any schema elements are removed. This is usually the phase in which custom application code is executed, to copy existing data from other tables and fields before they are dropped in the next phase. For example, if there is a new required 1--1 relation, the custom code might analyze the other data in the rows of that table to determine which value that row should have for the new foreign key. * DropSchema: Drop any unneeded schema elements and data * CreatePrimaryKeys: Create primary keys required by the schema. This includes both new primary keys as well as reestablishing primary keys that were temporarily dropped in the second phase. * CreateConstraintsAndIndexes: Create constraints and indexes required by the schema. This includes both new constraints and indexes as well as reestablishing constraints and indexes that were temporarily dropped in the second phase. * UpdateMetadata: Update the Quino-specific metadata table for the affected elements. [Executing the migration plan] The commands are then executed and the results logged. Afterward, the schema is imported again, to verify that there are no differences between the target model and the database. In some (always rarer) cases, there will still be differences, in which case, you can execute the new migration plan to repair those differences as well. In development, this works remarkably well and often, without further intervention. [Fixing failed migrations] In some cases, there is data in the database that, while compatible with the current database schema, is incompatible with the updated schema. This usually happens when a new property or constraint is introduced. For example, a new required property is added that does not have a default value or a new unique index is added which existing data violates. In these cases, there are two things that can be done: * Either the database data is cleaned up in a way that makes it compatible with the target schema [4] * Or the developer must add custom logic to the metadata elements involved. This usually means that the developer must set a default value on a property. In rarer cases, the developer must attach logic to the affected metadata (e.g. the property or index that is causing the issue) that runs during schema migration to create new data or copy it from elsewhere in order to ensure that constraints are satisfied when they are reestablished at the end of the migration. In general, it's strongly advised to perform a migration against a replica of the true target database (e.g. a production database) in order to guarantee that all potential data situations have been anticipated with custom code, if necessary. [Quino Migration versus EF Migrations] It's important to point out that Quino's schema migration is considerably different from that employed by EF (which it picked up from the Active Migrations in Ruby, often used with Ruby on Rails). In those systems, the developer generates specific migrations to move from one model version to another. There is a clear notion of upgrading versus downgrading. Quino only recognizes migrating from an arbitrary model to another arbitrary model. This makes Quino's migration exceedingly friendly when moving between development branches, unlike EF, whose "deficiencies in this area have been documented" . -------------------------------------------------------------------------------- [1] The default is to use only MetaIds. There is a mode in which identifiers are used as a fallback but it is used only for tools that import schemas that were not generated by Quino. Again, if the Quino metadata table hasn't been damaged, this strict form of mapping will work extremely well. [1] The Winform and Web user interfaces for Quino both include built-in feedback for interacting with the schema migration. There are also two standalone tools to migrate database schemas: a Winform application and a Windows console application. [1] The form of these commands is currently a mix of SQL and custom C# code. A future feature of the migration will be to have all commands available as SQL text so that the commands, instead of being executed directly, could be saved as a file and reviewed and executed by DBAs instead of letting the tool do it. We're not quite there yet, but proceeding nicely. [1] This is generally what a developer does with his or her local database. The data contained therein can usually be more or less re-generated. If there is a conflict during migration, a developer can determine whether custom code is necessary or can sometimes determine that the data situation that causes the problem isn't something that comes up in production anyway and just remove the offending elements or data until the schema migration succeeds. ]]> https://www.earthli.com/news/view_article.php?id=2978 2014-10-20T15:23:19+02:00 https://earthli.com/users/marco

" We use Microsoft Entity Framework (EF) Migrations in one of our... ]]>

" We use Microsoft Entity Framework (EF) Migrations in one of our projects where we are unable to use Quino. We were initially happy to be able to automate database-schema changes. After using it for a while, we have decidedly mixed feelings. As developers of our own schema migration for the Quino ORM, we're always on the lookout for new and better ideas to improve our own product. If we can't use Quino, we try to optimize our development process in each project to cause as little pain as possible. [EF Migrations and branches] We ran into problems in integrating EF Migrations into a development process that uses feature branches. As long as a developer stays on a given branch, there are no problems and EF functions relatively smoothly. [1] However, if a developer switches to a different branch -- with different migrations -- EF Migrations is decidedly less helpful. It is, in fact, quite cryptic and blocks progress until you figure out what's going on. Assume the following not-uncommon situation: * The project is created in the master branch * The project has an initial migration BASE * Developers A and B migrate their databases to BASE * Developer A starts branch feature/A and includes migration A in her database * Developer B starts branch feature/B and includes migration B in his database We now have the situation in which two branches have different code and each has its own database schema. Switching from one branch to another with Git quickly and easily addresses the code differences. The database is, unfortunately, a different story. Let's assume that developer A switches to branch feature/B to continue working there. The natural thing for A to do is to call "update-database" from the Package Manager Console [2]. This yields the following message -- all-too-familiar to EF Migrations developers. [image] "Unable to update database to match the current model because there are pending changes and automatic migration is disabled. Either write the pending changes to a code-based migration or enable automatic migration. [...]" This situation happens regularly when working with multiple branches. It's even possible to screw up a commit within a single branch, as illustrated in the following real-world example. * Add two fields to an existing class * Generate a migration with code that adds two fields * Migrate the database * Realize that you don't need one of the two fields * Remove the C# code from the migration for that field * Tests run green * Commit everything and push it As far as you're concerned, you committed a single field to the model. When your co-worker runs that migration, it will be applied, but EF Migrations immediately thereafter complains that there are pending model changes to make. How can that be? [Out-of-sync migrations != outdated database] Just to focus, we're actually trying to get real work done, not necessarily debug EF Migrations. We want to answer the following questions: 1. Why is EF Migrations having a problem updating the schema? 2. How do I quickly and reliably update my database to use the current schema if EF Migrations refuses to do it? The underlying reason why EF Migrations has problems is that it does not actually know what the schema of the database is. It doesn't read the schema from the database itself, but relies instead on a copy of the EF model that it stored in the database when it last performed a successful migration. That copy of the model is also stored in the resource file generated for the migration. EF Migrations does this so that the migration includes information about which changes it needs to apply and about the model to which the change can be applied. If the model stored in the database does not match the model stored with the migration that you're trying to apply, EF Migrations will not update the database. This is probably for the best, but leads us to the second question above: what do we have to do to get the database updated? [Generate a migration for those "pending changes"] The answer has already been hinted at above: we need to fix the model stored in the database for the last migration. Let's take a look at the situation above in which your colleague downloaded what you thought was a clean commit. From the Package Manager Console, run add-migration foo to scaffold a migration for the so-called "pending changes" that EF Migrations detected. That's interesting: EF Migrations thinks that your colleague should generate a migration to drop the column that you'd only temporarily added but never checked in. That is, the column isn't in his database, it's not in your database, but EF Migrations is convinced that it was once in the model and must be dropped. How does EF Migrations even know about a column that you added to your own database but that you removed from the code before committing? What dark magic is this? The answer is probably obvious: you did check in the change. The part that you can easily remove (the C# code) is only half of the migration. As mentioned above, the other part is a binary chunk stored in the resource file associated with each migration. These BLOBS are stored in the table _MigrationHistory table in the database. [image][image] [How to fix this problem and get back to work] Here's the tl;dr: generate a "fake" migration, remove all of the C# code that would apply changes to the database (shown below) and execute update-database from the Package Manager Console. [image] This may look like it does exactly nothing. What actually happens is that it includes the current state of the EF model in the binary data for the last migration applied to the database (because you just applied it). Once you've applied the migration, delete the files and remove them from the project. This migration was only generated to fix your local database; do not commit it. [Everything's cool now, right?] Applying the fix above doesn't mean that you won't get database errors. If your database schema does not actually match the application model, EF will crash when it assumes fields or tables are available which do not exist in your database. Sometimes, the only way to really clean up a damaged database -- especially if you don't have the code for the migrations that were applied there [3] -- is to remove the misapplied migrations from your database, undo all of the changes to the schema (manually, of course) and then generate a new migration that starts from a known good schema. [Conclusions and comparison to Quino] The obvious answer to the complaint "it hurts when I do this" is "stop doing that". We would dearly love to avoid these EF Migrations-related issues but developing without any schema-migration support is even more unthinkable. We'd have to create upgrade scripts manually or would have to maintain scripts to generate a working development database and this in each branch. When branches are merged, the database-upgrade scripts have to be merged and tested as well. This would be a significant addition to our development process, has maintainability and quality issues and would probably slow us down even more. And we're certainly not going to stop developing with branches, either. We were hoping to avoid all of this pain by using EF Migrations. That EF Migrations makes us think of going back to manual schema migration is proof that it's not nearly as elegant a solution as our own Quino schema migration, which never gave us these problems. Quino actually reads the schema in the database and compares that model directly against the current application model. The schema migrator generates a custom list of differences that map from the current schema to the desired schema and applies them. There is user intervention but it's hardly ever really required. This is an absolute godsend during development where we can freely switch between branches without any hassle. [4] Quino doesn't recognize "upgrade" versus "downgrade" but instead applies "changes". This paradigm has proven to be a much better fit for our agile, multi-branch style of development and lets us focus on our actual work rather than fighting with tools and libraries. -------------------------------------------------------------------------------- [1] EF Migrations as we use it is tightly bound to SQL Server. Just as one example, the inability of SQL Server to resolve cyclic cascade dependencies is in no way shielded by EF Migrations. Though the drawback originates in SQL Server, EF Migrations simply propagates it to the developer, even though it purports to provide an abstraction layer. Quino, on the other hand, does the heavy lifting of managing triggers to circumvent this limitation. [1] As an aside, this is a spectacularly misleading name for a program feature. It should just be called "Console". [1] I haven't ever been able to use the Downgrade method that is generated with each migration, but perhaps someone with more experience could explain how to properly apply such a thing. If that doesn't work, the method outlined above is your only fallback. [1] The aforementioned database-script maintenance or having only very discrete schema-update points or maintaining a database per branch and switching with configuration files or using database backups or any other schemes that end up distracting you from working. ]]> https://www.earthli.com/news/view_article.php?id=3038 2014-09-14T16:09:45+02:00 https://earthli.com/users/marco

but a lot of the content was in demonstrations on the command-line. 1. "Download the presentation" 2. Unzip to a local folder 3. Open index.html in a modern web browser ... ]]>

but a lot of the content was in demonstrations on the command-line. 1. "Download the presentation" 2. Unzip to a local folder 3. Open index.html in a modern web browser (Chrome/Opera/Firefox work the best; IE has some rendering issues) We learned a few very interesting things: * PowerShell is pre-installed on every modern Windows computer * You can PowerShell to other machines (almost like ssh!) * Windows developers should definitely learn how to use PowerShell. * Unix administrators who have to work on Windows machines should definitely learn how to use PowerShell. The underlying functionality of the operating system is much more discoverable via command line, get-command and get-member than the GUI. * You should definitely install ConEmu * When running ConEmu, make sure that you start a PowerShell session rather than the default Cmd session. * If you're writing scripts, you should definitely install and use the ISE, which is an IDE for PowerShell scripts with debugging, code-completion, lists of available commands and much better copy/paste than the standard console. * The "PowerShell Language Reference v3" is a very useful and compact reference for beginners and even for more advanced users [ConEmu Setup] The easiest way to integrate PowerShell into your workflow is to make it eminently accessible by installing ConEmu. ConEmu is a Windows command-line with a tabbed interface and offers a tremendous number of power-user settings and features. You can tweak it to your heart's content. [image]I set mine up to look like the one that Tymon had in the demonstrations (shown on my desktop to the right). 1. Download "ConEmu" ; I installed version 140814, the most recent version marked as "beta". There is no official release yet, but the software is quite mature. 2. Install it and run it. I didn't allow the Win + Num support because I know that I'd never use it. YMMV and you can always change your choice from the preferences. 3. Show the settings to customize your installation. There are a ton of settings, so I listed the ones I changed below. 4. [image]Set the window size to something a bit larger than the standard settings, especially if you have a larger monitor. I use 120 x 40. 5. [image]Choose the color scheme you want to use. I'm using the standard PowerShell colors but a lot of popular, darker schemes are also available (e.g. Monokai). 6. [image]Check out the hotkeys and set them up accordingly. The only key I plan on using is the one to show ConEmu. On the Swiss-German keyboard, it's Ctrl + ¨. 7. [image]The default console is not transparent, but there are those of us who enjoy a bit of transparency. Again, YMMV. I turned it on and left the slider at the default setting. 8. [image]And, finally, you can turn on Quake-style console mode to make it drop down from the top of your primary monitor instead of appearing in a free-floating window. -------------------------------------------------------------------------------- [1] and one former Encodo employee -- hey "Stephan" ! ]]> https://www.earthli.com/news/view_article.php?id=3024 2014-08-08T10:20:08+02:00 https://earthli.com/users/marco

i.OverdueByTwoWeeks) .SelectMany(i => i.GetHistoricalAssignees() .Select(a => new { a.FirstName, a.LastName }) ); foreach (var name in names) { Console.WriteLine("{1}, {0}", name.FirstName, name.LastName); } If either GetHistoricalAssignees() or GetOpenItems() might return null, then we'd have to write the code above as follows instead: var openItems = GetOpenItems(); if (openItems != null) { var names = openItems .Where(i => i.OverdueByTwoWeeks) .SelectMany(i => (i.GetHistoricalAssignees() ?? Enumerable.Empty()) .Select(a => new { a.FirstName, a.LastName }) ); foreach (var name in names) { Console.WriteLine("{1}, {0}", name.FirstName, name.LastName); } } This seems like exactly the kind of code we'd like to avoid writing, if possible. It's also the kind of code that calling clients are unlikely to write, which will lead to crashes with NullReferenceExceptions. As we'll see below, there are people that seem to think that's perfectly OK. I am not one of those people, but I digress. The post, "Is it Really Better to 'Return an Empty List Instead of null'? / Part 1" by Christian Neumanns serves as a good example of an article that seems to be providing information but is just trying to distract people into accepting it as a source of genuine information. He introduces his topic with the following vagueness. "If we read through related questions in Stackoverflow and other forums, we can see that not all people agree. There are many different, sometimes truly opposite opinions. For example, the top rated answer in the Stackoverflow question Should functions return null or an empty object? (related to objects in general, not specifically to lists) tells us exactly the opposite: "Returning null is usually the best idea ..." The statement "we can see that not all people agree" is a tautology. I would split the people into groups of those whose opinions we should care about and everyone else. The statement "There are many different, sometimes truly opposite opinions" is also tautological, given the nature of the matter under discussion -- namely, a question that can only be answered as "yes" or "no". Such questions generally result in two camps with diametrically opposed opinions. As the extremely long-winded pair of articles writes: sometimes you can't be sure of what an external API will return. That's correct. You have to protect against those with ugly, defensive code. But don't use that as an excuse to produce even more methods that may return null. Otherwise, you're just part of the problem. The second article "Is it Really Better to 'Return an Empty List Instead of null'? - Part 2" by Christian Neumanns includes many more examples. I just don't know what to say about people that write things like "Bugs that cause NullPointerExceptions are usually easy to debug because the cause and effect are short-distanced in space (i.e. location in source code) and time." While this is kind of true, it's also even more true that you can't tell the difference between such an exception being caused by a savvy programmer who's using it to his advantage and a non-savvy programmer whose code is buggy as hell. He has a ton of examples that try to distinguish between a method that returns an empty sequence being different from a method that cannot properly answer a question. This is a concern and a very real distinction to make, but the answer is not to return null to indicate nonsensical input. The answer is to throw an exception. The method providing the sequence should not be making decisions about whether an empty sequence is acceptable for the caller. For sequences that cannot logically be empty, the method should throw an exception instead of returning null to indicate "something went wrong". A caller may impart semantic meaning to an empty result and also throw an exception (as in his example with a cycling team that has no members). If the display of such a sequence on a web page is incorrect, then that is the fault of the caller, not of the provider of the sequence. * If data is not yet available, but should be, throw an exception * If data is not available but the provider isn't qualified to decide, return an empty sequence * If the caller receives an empty sequence and knows that it should not be empty, then it is responsible for indicating an error. That there exists calling code that makes assumptions about return values that are incorrect is no reason to start returning values that will make calling code crash with a NullPointerException. All of his examples are similar: he tries to make the pure-data call to retrieve a sequence of elements simultaneously validate some business logic. That's not a good idea. If this is really necessary, then the validity check should go in another method. The example he cites for getting the amount from a list of PriceComponents is exactly why most aggregation functions in .NET throw an exception when the input sequence is empty. But that's a much better way of handling it -- with a precise exception -- than by returning null to try to force an exception somewhere in the calling code. But the upshot for me is: I am not going to write code that, when I call it, forces me to litter other code with null-checks. That's just ridiculous. ]]> https://www.earthli.com/news/view_article.php?id=3006 2014-08-08T10:20:05+02:00 https://earthli.com/users/marco

" two articles" , we managed to reduce the number of queries executed when opening the calendar of Encodo's time-tracking product Punchclock from one very slow query per person to a single very fast query. Because we're talking about latency in these articles, we'd also like to... ]]>

" two articles" , we managed to reduce the number of queries executed when opening the calendar of Encodo's time-tracking product Punchclock from one very slow query per person to a single very fast query. Because we're talking about latency in these articles, we'd also like to clear away a few other queries that aren't related to time entries but are still wasting time. [Lazy-loading unneeded values] In particular, the queries that "Load values" for person objects look quite suspicious. These queries don't take a lot of time to execute but they will definitely degrade performance in high-latency networks. [1] [image] As we did before, we can click on one of these queries to show the query that's being loaded. In the screenshot below, we see that the person's picture is being loaded for each person in the drop-down list. [image] We're not showing pictures in the drop-down list, though, so this is an extravagant waste of time. On a LAN, we hardly notice how wasteful we are with queries; on a WAN, the product will feel...sluggish. [What is a load-group?] In order to understand the cause of these queries, you must first know that Quino allows a developer to put metadata properties into different load-groups. A load-group has the following behavior: If the value for a property in a load-group is requested on an object, the values for all of the properties in the load-group are retrieved with a single query and set on that object. The default load-group of an object's metadata determine the values that are initially retrieved and applied to objects materialized by the ORM. The metadata for a person puts the "picture" property of a person into a separate load-group so that the value is not loaded by default when people objects are loaded from the data driver. This is a good balance because business logic will avoid downloading a lot of unwanted picture data by default. Business logic that needs the pictures can either explicitly include the picture in the query or let the value be lazy-loaded by the ORM when it is accessed. The proper solution depends on the situation. [Lazy-loaded property values] As before, we can check the stack trace of the query to figure out which application component is triggering the call. In this case, the culprit is the binding list that we are using to attach the list of people to the drop-down control. The binding list binds the values for all of the properties in a metaclass (e.g. "person"), triggering a lazy load when it accesses the "picture" property. To avoid the lazy-load, we can create a wrapper of the default metadata for a person and remove/hide the property so that the binding list will no longer access it. This is quite easy [2], as shown in the code below. var personMetaClass = new WrapMetaClass(Person.Metadata); personMetaClass.Properties.Remove(Person.MetaProperties.Picture); var query = new Query(personMetaClass); With this simple fix, the binding list no longer knows about the picture property, doesn't retrieve values for that property and therefore no longer triggers any queries to lazily load the pictures from the database for each person object. The screenshot of the statistics window below shows up that we were successful. We have two main queries: one for the list of people to show in the dropdown control and one for the time entries to show in the calendar. [image] [Final version] For completeness, here's the code that Punchclock is using in the current version of Quino (1.11). var personMetaClass = new WrapMetaClass(Person.Metadata); personMetaClass.Properties.Remove(Person.MetaProperties.Picture); var accessToolkit = new PostgreSqlMetaDatabase().AccessToolkit; var query = new Query(personMetaClass); query.CustomCommandText = new CustomCommandText(); query.CustomCommandText.SetSection( CommandTextSections.Where, CommandTextAction.Replace, string.Format( "EXISTS (SELECT id FROM {0} WHERE {1} = {2})", accessToolkit.GetName(TimeEntry.Metadata), accessToolkit.GetField(TimeEntry.MetaProperties.PersonId), accessToolkit.GetField(Person.MetaProperties.Id) )> ); var people = Session.GetList(query); [Future, improved version] Once we fix the but in the WhereExists join type mentioned in the previous article and add the fluent methods for constructing wrappers mentioned in the footnote below, the code will be as follows: var personMetaClass = Person.Metadata. Wrap(). RemoveProperty(Person.MetaProperties.Picture); var accessToolkit = new PostgreSqlMetaDatabase().AccessToolkit; var people = Session.GetList( new Query(personMetaClass). Join(Person.MetaRelations.TimeEntries, JoinType.WhereExists). Query ); This concludes our investigation into performance issues with Quino and Punchclock. -------------------------------------------------------------------------------- [1] You may have noticed that these calls to "load values" are technically lazy-loaded but don't seem to be marked as such in the screenshots. This was a bug in the statistics viewer that I discovered and addressed while writing this article. [image] [1] This is a rather old API and hasn't been touched with the "fluent" wand that we've applied in other parts of the Quino API. A nicer way of writing it would be to create an extension methods called Wrap() and RemoveProperty that return the wrapper class, like so: var personMetaClass = Person.Metadata. Wrap(). RemoveProperty(Person.MetaProperties.Picture); var query = new Query(personMetaClass); But that will have to wait for a future version of Quino. ]]> https://www.earthli.com/news/view_article.php?id=3005 2014-07-04T09:09:05+02:00 https://earthli.com/users/marco

, we partially addressed a performance problem in the calendar of Encodo's time-tracking product, Punchclock. While we managed to drastically reduce the amount of time taken by each query (>95% time saved), we were still executing more queries than strictly necessary. The... ]]>

, we partially addressed a performance problem in the calendar of Encodo's time-tracking product, Punchclock. While we managed to drastically reduce the amount of time taken by each query (>95% time saved), we were still executing more queries than strictly necessary. The query that we're trying to optimized further is shown below. var people = Session.GetList(). Where(p => Session.GetCount(p.TimeEntries.Query) > 0). ToList(); This query executes one query to get all the people and then one query per person to get the number of time entries per person. Each of these queries by itself is very fast. High latency will cause them to be slow. In order to optimize further, there's really nothing for it but to reduce the number of queries being executed. Let's think back to what we're actually trying to accomplish: We want to get all people who have at least one time entry. Can't we get the database to do that for us? Some join or existence check or something? How about the code below? var people = Session.GetList( Session.CreateQuery(). Join(Person.MetaRelations.TimeEntries, JoinType.WhereExists). Query ); What's happening in the code above? We're still getting a list of people but, instead of manipulating the related TimeEntries for each person locally, we're joining the TimeEntries relation with the Quino query Join() method and changing the join type from the default All to the restrictive WhereExists. This sounds like exactly what we want to happen! There is no local evaluation or manipulation with Linq and, with luck, Quino will be able to map this to a single query on the database. This is the best possible query: it's purely declarative and will be executed as efficiently as the back-end knows how. There's just one problem: the WhereExists join type is broken in Quino 1.11. Never fear, though! We can still get it to work, but we'll have to do a bit of work until the bug is fixed in Quino 1.12. The code below builds on lessons learned in the earlier article, "Mixing your own SQL into Quino queries: part 2 of 2" to use custom query text to create the restriction instead of letting Quino do it. var accessToolkit = new PostgreSqlMetaDatabase().AccessToolkit; var query = Session.CreateQuery(); query.CustomCommandText = new CustomCommandText(); query.CustomCommandText.SetSection( CommandTextSections.Where, CommandTextAction.Replace, string.Format( "EXISTS (SELECT id FROM {0} WHERE {1} = {2})", accessToolkit.GetName(TimeEntry.Metadata), accessToolkit.GetField(TimeEntry.MetaProperties.PersonId), accessToolkit.GetField(Person.MetaProperties.Id) ) ); var people = Session.GetList(query); A look at the statistics is very encouraging: [image] We're down to one 29ms query for the people and an even quicker query for all the relevant time entries. [1] We can see our query text appears embedded in the SQL generated by Quino, just as we expected. There are a few other security-related queries that execute very quickly and hardly need optimization. We've come much farther in this article and we're almost done. In the next article, we'll quickly clean up a few other queries that are showing up in the statistics and that have been nagging us since the beginning. -------------------------------------------------------------------------------- [1] The time-entry query is not representative because my testing data set didn't include time entries for the current day and I was too lazy to page around to older data. ]]> https://www.earthli.com/news/view_article.php?id=3004 2014-06-27T10:07:40+02:00 https://earthli.com/users/marco

, we discussed a performance problem in the calendar of Encodo's time-tracking product, Punchclock. Instead of guessing at the problem, we profiled the application using the database-statistics window available to all Quino applications. [1] We quickly discovered that most of... ]]>

, we discussed a performance problem in the calendar of Encodo's time-tracking product, Punchclock. Instead of guessing at the problem, we profiled the application using the database-statistics window available to all Quino applications. [1] We quickly discovered that most of the slowdown stems from the relatively innocuous line of code shown below. var people = Session.GetList(). Where(p => p.TimeEntries.Any()). ToList(); [First things first: what does the code do?] Before doing anything else, we should establish what the code does. Logically, it retrieves a list of people in the database who have recorded at least one time entry. The first question we should ask at this point is: does the application even need to do this? The answer in this case is 'yes'. The calendar includes a drop-down control that lets the user switch between the calendars for different users. This query returns the people to show in this drop-down control. With the intent and usefulness of the code established, let's dissect how it is accomplishing the task. 1. The Session.GetList() portion retrieves a list of all people from the database 2. The Where() method is applied locally for each object in the list [2] 3. For a given person, the list of TimeEntries is accessed 4. This access triggers a lazy load of the list 5. The Any() method is applied to the full list of time entries 6. The ToList() method creates a list of all people who match the condition Though the line of code looks innocuous enough, it causes a huge number of objects to be retrieved, materialized and retained in memory -- simply in order to check whether there is at least one object. This is a real-world example of a performance problem that can happen to any developer. Instead of blaming the developer who wrote this line of code, its more important to stay vigilant to performance problems and to have tools available to quickly and easily find them. [Stop creating all of the objects] The first solution I came up with [3] was to stop creating objects that I didn't need. A good way of doing this and one that was covered in "Quino: partially-mapped queries" is to use cursors instead of lists. Instead of using the generated list TimeEntries, the following code retrieves a cursor on that list's query and materializes at most one object for the sub-query. var people = Session.GetList().Select(p => { using (var cursor = Session.CreateCursor(p.TimeEntries.Query)) [4] { return cursor.Any(); } }).ToList(); A check of the database statistics shows improvement, as shown below. [image] Just by using cursors, we've managed to reduce the execution time for each query by about 75%. [5] Since all we're interested in finding out is whether there is at least one time entry for a person, we could also ask the database to count objects rather than to return them. That should be even faster. The following code is very similar to the example above but, instead of getting a cursor based on the TimeEntries query, it gets the count. var people = Session.GetList(). Where(p => Session.GetCount(p.TimeEntries.Query) > 0). ToList(); How did we do? A check of the database statistics shows even more improvement, as shown below. [image] We're now down to a few dozen milliseconds for all of our queries, so we're done, right? A 95% reduction in query-execution time should be enough. Unfortunately, we're still executing just as many queries as before, even though we're taking far less time to execute them. This is better, but still not optimal. In high-latency situations, the user is still likely to experience a significant delay when opening the calendar since each query's execution time is increased by the latency of the connection. In a local network, the latency is negligible; on a WAN, we still have a problem. In the next article, we’ll see if we can't reduce the number of queries being executed. -------------------------------------------------------------------------------- [1] This series of articles shows the statistics window as it appears in Winforms applications. The data-provider statistics are also available in Quino web applications as a Glimpse plug-in. [1] It is important for users of the Microsoft Entity Framework (EF) to point out that Quino does not have a Linq-to-Sql mapper. That means that any Linq expressions like Where() are evaluated locally instead of being mapped to the database. There are various reasons for this but the main one is that we ended up preferring a strict boundary between the mappable query API and the local evaluation API. Anything formulated with the query API is guaranteed to be executed by the data provider (even if it must be evaluated locally) and anything formulated with Linq is naturally evaluated locally. In this way, the code is clear in what is sent to the server and what is evaluated locally. Quino only very, very rarely issues an "unmappable query" exception, unlike EF, which occasionally requires contortions until you've figured out which C# formulation of a particular expression can be mapped by EF. [1] Well, the first answer I'm going to pretend I came up with. I actually thought of another answer first, but then quickly discovered that Quino wasn't mapping that little-used feature correctly. I added an issue to tackle that problem at a later date and started looking for workarounds. That fix will be covered in the next article in this series. [1] Please note that cursors are disposable and that the calling application is responsible for cleanup. Failure to dispose of a cursor that has been at least partially iterated will result in an open connection in the underlying database providers associated with the query and will eventually lead to connection-pool exhaustion on those databases. [1] Please ignore the fact that we also dropped 13 person queries. This was not due to any fix that we made but rather that I executed the test slightly differently...and was too lazy to make a new screenshot. The 13 queries are still being executed and we'll tackle those in the last article in this series. ]]> https://www.earthli.com/news/view_article.php?id=3003 2014-06-20T10:44:29+02:00 https://earthli.com/users/marco

().Where(p => p.TimeEntries.Any()).ToList(); This looks like a nicely declarative way of getting data, but to the trained eye of a Quino developer, it's clear what the problem is. In the next couple of articles, we'll take a closer look at what exactly the problem is and how we can improve the speed of this query. We'll also take a look at how we can improve the Quino query API to make it harder for code like the line above to cause performance problems. -------------------------------------------------------------------------------- [1] Encodo just turned nine years old, but we used a different time-entry system for the first couple of years. If you're interested in our time-entry software history, here it is: 1. 06.2005 -- Start off with Open Office spreadsheets 2. 04.2007 -- Switch to a home-grown, very lightweight time tracker based on an older framework we'd written (Punchclock 1.0) 3. 08.2008 -- Start development of Quino 4. 04.2010 -- Initial version of Punchclock 2.0; start dogfooding Quino ]]> https://www.earthli.com/news/view_article.php?id=2997 2014-06-18T08:10:36+02:00 https://earthli.com/users/marco

, we listed a lot of questions that you should continuously ask yourself when you're writing code. Even when you think you're not designing anything, you're actually making decisions that will affect either other team members or future versions of you. In particular, we'd... ]]>

, we listed a lot of questions that you should continuously ask yourself when you're writing code. Even when you think you're not designing anything, you're actually making decisions that will affect either other team members or future versions of you. In particular, we'd like to think about how we can reconcile a development process that involves asking so many questions and taking so many facets into consideration with YAGNI. [Designing != Implementing] The implication of this principle is, that if you aren't going to need something, then there's no point in even thinking about it. While it's absolutely commendable to adopt a YAGNI attitude, not building something doesn't mean not thinking about it and identifying potential pitfalls. A feature or design concept can be discussed within a time-box. Allocate a fixed, limited amount of time to determine whether the feature or design concept needs to be incorporated, whether it would be nice to incorporate it or possibly to jettison it if it's too much work and isn't really necessary. The overwhelming majority of time wasted on a feature is in the implementation, debugging, testing, documentation and maintenance of it, not in the design. Granted, a long design phase can be a time-sink -- especially a "perfect is the enemy of the good" style of design where you're completely blocked from even starting work. With practice, however, you'll learn how to think about a feature or design concept (e.g. extensibility) without letting it ruin your schedule. If you don't try to anticipate future needs at all while designing your API, you may end up preventing that API from being extended in directions that are both logical and could easily have been anticipated. If the API is not extensible, then it will not be used and may have to be rewritten in the future, losing more time at that point rather than up front. This is, however, only a consideration you must make. It's perfectly acceptable to decide that you currently don't care at all and that a feature will have to be rewritten at some point in the future. You can't do this kind of cost-benefit analysis and risk-management if you haven't taken time to identify the costs, benefits or risks. [Document your process] At Encodo, we encourage the person who's already spent time thinking about this problem to simply document the drawbacks and concessions and possible ideas in an issue-tracker entry that is linked to the current implementation. This allows future users, maintainers or extenders of the API to be aware of the thought process that underlies a feature. It can also help to avoid misunderstandings about what the intended audience and coverage of an API are. The idea is to eliminate assumptions. A lot of time can be wasted when maintenance developers make incorrect assumptions about the intent of code. If you don't have time to do any of this, then you can write a quick note in a task list that you need to more fully document your thoughts on the code you're writing. And you should try to do that soon, while the ideas are still relatively fresh in your mind. If you don't have time to think about what you're doing even to that degree, then you're doing something wrong and need to get organized better. That is, you if you can't think about the code you're writing and don't have time to document your process, even minimally, then you shouldn't be writing that code. Either that, or you implicitly accept that others will have to clean up your mess. And "others" includes future versions of you. (E.g. the you who, six months from now, is muttering, "who wrote this crap?!?") [Be Honest about Hacking] As an example, we can consider how we go from a specific feature in the context of a project to thinking about where the functionality could fit in to a suite of products -- that may or may not yet exist. And remember, we're only thinking about these things. And we're thinking about them for a limited time -- a time-box. You don't want to prevent your project from moving forward, but you also don't want to advance at all costs. Advancing in an unstructured way is called hacking and, while it can lead to a short-term win, it almost always leads to short-to-medium term deficits. You can still write code that is hacked and looks hacked, if that is the highest current priority, but you're not allowed to forget that you did so. You must officially designate what you're doing as a hot-zone of hacking so that the Hazmat team can clean it up later, if needed. A working prototype that is hacked together just so it works for the next demonstration is great as long as you don't think that you can take it into production without doing the design and documentation work that you initially skipped. If you fail to document the deficits that prevent you from taking a prototype to production, then how will you address those deficits? It will cost you much more time and pain to determine the deficits after the fact. Not only that, but unless you do a very good job, it is your users that will most likely be finding deficits -- in the form of bugs. If your product is just a hacked mess of spaghetti code with no rhyme or reason, another developer will be faster and produce more reliable code by just starting over. Trying to determine the flaws, drawbacks and hacks through intuition and reverse-engineering is slower and more error-prone than just starting with a clean slate. Developers on such a project will not be able to save time -- and money -- by building on what you've already made. [A note on error-handling] Not to be forgotten is a structured approach to error-handling. The more "hacked" the code, the more stringent the error-checking should be. If you haven't had time yet to write or test code sufficiently, then that code shouldn't be making broad decisions about what it thinks are acceptable errors. Fail early, fail often. Don't try to make a hacked mess of code bullet-proof by catching all errors in an undocumented manner. Doing so is deceptive to testers of the product as well as other developers. If you're building a demo, make sure the happy path works and stick to it during the demo. If you do have to break this rule, add the hacks to a demo-specific branch of the code that will be discarded later. [Working with a documented project] If, however, the developer can look at your code and sees accompanying notes (either in an issue tracker, as TODOs in the code or some other form of documentation), that developer knows where to start fixing the code to bring it to production quality. For example, it's acceptable to configure an application in code as long as you do it in a central place and you document that the intent is to move the configuration to an external source when there's time. If a future developer finds code for support for multiple database connections and tests that are set to ignore with a note/issue that says "extend to support multiple databases", that future developer can decide whether to actually implement the feature or whether to just discard it because it has been deprecated as a requirement. Without documentation or structure or an indication which parts of the code were thought-through and which are considered to be hacked, subsequent developers are forced to make assumptions that may not be accurate. They will either assume that hacked code is OK or that battle-tested code is garbage. If you don't inform other developers of your intent when your're writing the code -- best done with documentation, tests and/or a cleanly designed API -- then it might be discarded or ignored, wasting even more time and money. If you're on a really tight time-budget and don't have time to document your process correctly, then write a quick note that you think the design is OK or the code is OK, but tell your future self or other developers what they're looking at. It will only take you a few minutes and you'll be glad you did -- and so will they. ]]> https://www.earthli.com/news/view_article.php?id=2996 2014-06-03T10:25:46+02:00 https://earthli.com/users/marco

, which seems to have less to do with programming Erlang and much more to do with programming properly. The advice contained in it seems to be only for Erlang programmers, but the idea of strictly enforcing APIs between software components is neither new nor language-specific. ]]> https://www.earthli.com/news/view_article.php?id=2994 2014-05-31T08:55:13+02:00 https://earthli.com/users/marco

for a more in-depth analysis, but that's the gist of it. This issue is still present in the ClientBase implementation in .NET 4.5.1. The linked article shows how you can add your own implementation of the Dispose() method in each generated client. An alternative is to use a generic adaptor if you don't feel like adding a custom dispose to every client you create. [1] public class SafeClient : IDisposable where T : ICommunicationObject, IDisposable { public SafeClient(T client) { if (client == null) { throw new ArgumentNullException("client"); } Client = client; } public T Client { get; private set; } public void Dispose() { Dispose(true); GC.SuppressFinalize(this); } protected virtual void Dispose(bool disposing) { if (disposing) { if (Client != null) { if (Client.State == CommunicationState.Faulted) { Client.Abort(); } else { Client.Close(); } Client = default(T); } } } } To use your WCF client safely, you wrap it in the class defined above, as shown below. using (var safeClient = new SafeClient(new SystemLoginServiceClient(...))) { var client = safeClient.Client; // Work with "client" } If you can figure out how to initialize your clients without passing parameters to the constructor, you could slim it down by adding a "new" generic constraint to the parameter T in SafeClient and then using the SafeClient as follows: using (var safeClient = new SafeClient()) { var client = safeClient.Client; // Work with "client" } -------------------------------------------------------------------------------- [1] The code included in this article is a sketch of a solution and has not been tested. It does compile, though. ]]> https://www.earthli.com/news/view_article.php?id=2995 2014-05-31T08:55:09+02:00 https://earthli.com/users/marco

Code 400: * The requested URI does not represent any resource on the server. * One of the request inputs is out of range. * One of the request inputs is not valid. * A required query parameter was not specified for this request. * One of the query parameters specified in the request URI is not supported. * An invalid value was specified for one of the query parameters in the request URI. Code 500: * The server encountered an internal error. Please retry the request. * The operation could not be completed within the permitted time. * The server is currently unable to receive requests. Please retry your request. "Twitter Error Codes & Responses" Code 400: "The request was invalid or cannot be otherwise served. An accompanying error message will explain further." Code 500: "Something is broken. Please post to the group so the Twitter team can investigate." "REST API Tutorial HTTP Status Codes" Code 400: "General error when fulfilling the request would cause an invalid state. Domain validation errors, missing data, etc. are some examples." Code 500: "A generic error message, given when no more specific message is suitable. The general catch-all error when the server-side throws an exception. Use this only for errors that the consumer cannot address from their end—never return this intentionally." "REST HTTP status codes" "For input validation failure: 400 Bad Request + your optional description. This is suggested in the book "RESTful Web Services"." ]]> https://www.earthli.com/news/view_article.php?id=2987 2014-04-17T21:30:02+02:00 https://earthli.com/users/marco

, we covered the basics of mixing custom SQL with ORM-generated queries. We also took a look at a solution that uses direct ADO database access to perform arbitrarily complex queries. In this installment, we will see more elegant techniques that make use of the CustomCommandText... ]]>

, we covered the basics of mixing custom SQL with ORM-generated queries. We also took a look at a solution that uses direct ADO database access to perform arbitrarily complex queries. In this installment, we will see more elegant techniques that make use of the CustomCommandText property of Quino queries. We'll approach the desired solution in steps, proceeding from attempt #1 -- attempt #"" <>5. tl;dr: Skip to attempt #5 to see the final result without learning why it's correct. [Attempt #1: Replacing the entire query with custom SQL] An application can assign the CustomCommandText property of any Quino query to override some of the generated SQL. In the example below, we override all of the text, so that Quino doesn't generate any SQL at all. Instead, Quino is only responsible for sending the request to the database and materializing the objects based on the results. [Test] public void TestExecuteCustomCommand() { var people = Session.GetList(); people.Query.CustomCommandText = new CustomCommandText { Text = @" SELECT ALL ""punchclock__person"".""id"", ""punchclock__person"".""companyid"", ""punchclock__person"".""contactid"", ""punchclock__person"".""customerid"", ""punchclock__person"".""initials"", ""punchclock__person"".""firstname"", ""punchclock__person"".""lastname"", ""punchclock__person"".""genderid"", ""punchclock__person"".""telephone"", ""punchclock__person"".""active"", ""punchclock__person"".""isemployee"", ""punchclock__person"".""birthdate"", ""punchclock__person"".""salary"" FROM punchclock__person WHERE lastname = 'Rogers'" }; Assert.That(people.Count, Is.EqualTo(9)); } This example solves two of the three problems outlined above: * It uses only a single query. * It will work with a remote application server (although it makes assumptions about the kind of SQL expected by the backing database on that server). * But it is even more fragile than the previous example as far as hard-coded SQL goes. You'll note that the fields expected by the object-materializer have to be explicitly included in the correct order. Let's see if we can address the third issue by getting Quino to format the SELECT clause for us. [Attempt #2: Generating the SELECT clause] The following example uses the AccessToolkit of the IQueryableDatabase to format the list of properties obtained from the metadata for a Person. The application no longer makes assumptions about which properties are included in the select statement, what order they should be in or how to format them for the SQL expected by the database. [Test] public virtual void TestExecuteCustomCommandWithStandardSelect() { var people = Session.GetList(); var accessToolkit = DefaultDatabase.AccessToolkit; var properties = Person.Metadata.DefaultLoadGroup.Properties; var fields = properties.Select(accessToolkit.GetField); people.Query.CustomCommandText = new CustomCommandText { Text = string.Format( @"SELECT ALL {0} FROM punchclock__person WHERE lastname = 'Rogers'", fields.FlattenToString() ) }; Assert.That(people.Count, Is.EqualTo(9)); } This example fixes the problem with the previous one but introduces a new problem: it no longer works with a remote application because it assumes that the client-side driver is a database with an AccessToolkit. The next example addresses this problem. [Attempt #3: Using a hard-coded AccessToolkit] The version below uses a hard-coded AccessToolkit so that it doesn't rely on the external data driver being a direct ADO database. It still makes an assumption about the database on the server but that is usually quite acceptable because the backing database for most applications rarely changes. [1] [Test] public void TestCustomCommandWithPostgreSqlSelect() { var people = Session.GetList(); var accessToolkit = new PostgreSqlMetaDatabase().AccessToolkit; var properties = Person.Metadata.DefaultLoadGroup.Properties; var fields = properties.Select(accessToolkit.GetField); people.Query.CustomCommandText = new CustomCommandText { Text = string.Format( @"SELECT ALL {0} FROM punchclock__person WHERE lastname = 'Rogers'", fields.FlattenToString() ) }; Assert.That(people.Count, Is.EqualTo(9)); } We now have a version that satisfies all three conditions to a large degree. The application uses only a single query and the query works with both local databases and remoting servers. It still makes some assumptions about database-schema names (e.g. "punchclock__person" and "lastname"). Let's see if we can clean up some of these as well. [Attempt #4: Replacing only the where clause] Instead of replacing the entire query text, an application can replace individual sections of the query, letting Quino fill in the rest of the query with its standard generated SQL. An application can append or prepend text to the generated SQL or replace it entirely. Because the condition for our query is so simple, the example below replaces the entire WHERE clause instead of adding to it. [Test] public void TestCustomWhereExecution() { var people = Session.GetList(); people.Query.CustomCommandText = new CustomCommandText(); people.Query.CustomCommandText.SetSection( CommandTextSections.Where, CommandTextAction.Replace, "lastname = 'Rogers'" ); Assert.That(people.Count, Is.EqualTo(9)); } That's much nicer -- still not perfect, but nice. The only remaining quibble is that the identifier lastname is still hard-coded. If the model changes in a way where that property is renamed or removed, this code will continue to compile but will fail at run-time. This is a not insignificant problem if your application ends up using these kinds of queries throughout its business logic. [Attempt #5: Replacing the where clause with generated field names] In order to fix this query and have a completely generic query that fails to compile should anything at all change in the model, we can mix in the technique that we used in attempts #2 and #3: using the AccessToolkit to format fields for SQL. To make the query 100% statically checked, we'll also use the generated metadata -- LastName -- to indicate which property we want to format as SQL. [Test] public void TestCustomWhereExecution() { var people = Session.GetList(); var accessToolkit = new PostgreSqlMetaDatabase().AccessToolkit; var lastNameField = accessToolkit.GetField(Person.MetaProperties.LastName); people.Query.CustomCommandText = new CustomCommandText(); people.Query.CustomCommandText.SetSection( CommandTextSections.Where, CommandTextAction.Replace, string.Format("{0} = 'Rogers'", lastNameField) ); Assert.That(people.Count, Is.EqualTo(9)); } The query above satisfies all of the conditions we outlined above. it's clear that the condition is quite simple and that real-world business logic will likely be much more complex. For those situations, the best approach is to fall back to using the direct ADO approach mixed with using Quino facilities like the AccessToolkit as much as possible to create a fully customized SQL text. Many thanks to "Urs" for proofreading and suggestions on overall structure. -------------------------------------------------------------------------------- [1] If an application needs to be totally database-agnostic, then it will need to do some extra legwork that we won't cover in this post. ]]> https://www.earthli.com/news/view_article.php?id=2985 2014-04-13T17:38:59+02:00 https://earthli.com/users/marco

(); people.Query.Where(Person.MetaProperties.Id, ExpressionOperator.In, ids); Assert.That(people.Count, Is.EqualTo(9)); } The ADO-access code is hidden inside the call to GetIds(), the implementation for which is shown below. Your application can get the connection for a session as described above and then create commands using the same helper class. If you call CreateCommand() directly on the ADO connection, you'll have a problem when running inside a transaction on SQL Server. The SQL Server ADO implementation requires that you assign the active transaction object to each command. Quino takes care of this bookkeeping for you if you use the helper method. private IEnumerable GetIds() { using (var helper = AdoDataConnectionTools.GetAdoConnection(Session, "Name")) { using (var command = helper.CreateCommand()) { command.AdoCommand.CommandText = @"SELECT id FROM punchclock__person WHERE lastname = 'Rogers'"; using (var reader = command.AdoCommand.ExecuteReader()) { while (reader.Read()) { yield return reader.GetInt32(0); } } } } } There are a few drawbacks to this approach: * Your application will make two queries instead of one. * The hard-coded SQL will break if you make model changes that affect those tables and fields. * The ADO approach only works if the application has a direct connection to the database. An application that uses ADO will not be able to switch to an application-server driver without modification. In the second part, we will improve on this approach by using the CustomCommandText property of a Quino query. This will allow us to use only a single query. We will also improve maintainability by reducing the amount of code that isn't checked by the compiler (e.g. the SQL text above). Stay tuned for part 2, coming soon! Many thanks to "Urs" for proofreading and suggestions on overall structure. -------------------------------------------------------------------------------- [1] This article uses features of Quino that will only become available in version 1.12. Almost all of the examples will also work in earlier versions but the AdoDataConnectionTools is not available until 1.12. The functionality of this class can, however, be back-ported if necessary. [1] More likely, though, is that the Quino schema migration will be prevented from applying updates if there are custom stored procedures that use tables and columns that need to be changed. Mixing your own SQL into Quino queries: part 1 of 2 ]]> https://www.earthli.com/news/view_article.php?id=2977 2014-03-28T15:53:54+01:00 https://earthli.com/users/marco

and was distraught/pleased [2] to discover that Java had in fact gotten two presents that C# doesn't already have. As you'll see, these two features aren't huge and the lack of them doesn't significantly impact design or expressiveness, but you know how jealousy works: Jealousy doesn't care. Jealousy is. I'm sure I'll get over it, but it will take time. [3] [Default methods and static interface methods] Java 8 introduces support for static methods on interfaces as well as default methods that, taken together, amount to functionality that is more or less what extensions methods brings to C#. In Java 8, you can define static methods on an interface, which is nice, but it becomes especially useful when combined with the keyword default on those methods. As defined in "Default Methods" : "Default methods enable you to add new functionality to the interfaces of your libraries and ensure binary compatibility with code written for older versions of those interfaces." In Java, you no longer have to worry that adding a method to an interface will break implementations of that interface in other jar files that have not yet been recompiled against the new version of the interface. You can avoid that by adding a default implementation for your method. This applies only to those methods where a default implementation is possible, of course. The page includes an example but it's relatively obvious what it looks like: public interface ITransformer { string Adjust(string value); string NewAdjust(string value) { return value.Replace(' ', '\t'); } } How do these compare with extension methods in C#? Extension methods are nice because they allow you to quasi-add methods to an interface without requiring an implementor to actually implement them. My rule of thumb is that any method that can be defined purely in terms of the public API of an interface should be defined as an extension method rather than added to the interface. Java's default methods are a twist on this concept that addresses a limitation of extension methods. What is that limitation? That the method definition in the extension method can't be overridden by the actual implementation behind the interface. That is, the default implementation can be expressed purely in terms of the public interface, but perhaps a specific implementor of the interface would like to do that plus something more. Or would perhaps like to execute the extension method in a different way, but only for a specific implementation. There is no way to do this with extension methods. Interface default methods in Java 8 allow you to provide a fallback implementation but also allows any class to actually implement that method and override the fallback. [Functional Interfaces] Functional interfaces are a nice addition, too, and something I've wanted in C# for some time. "Eric Meijer" of Microsoft doesn't miss an opportunity to point out that this is a must for functional languages (he's exaggerating, but the point is taken). Saying that a language supports functional interface simply means that a lambda defined in that language can be assigned to any interface with a single method that has the same signature as that lambda. An example in C# should make things clearer: public interface ITransformer { string Adjust(string value); } public static class Utility { public static void WorkOnText(string text, ITransformer) { // Do work } } In order to call WorkOnText() in C#, I am required to define a class that implements ITransformer. There is no other way around it. However, in a language that allows functional interfaces, I could call the method with a lambda directly. The following code looks like C# but won't actually compile. Utility.WorkOnText( "Hello world", s => s.Replace("Hello", "Goodbye cruel") ); For completeness, let's also see how much extra code it is do this in C#, which has no functional interfaces. public class PessimisticTransformer : ITransformer { public string Adjust(string value) { return value.Replace("Hello", "Goodbye cruel"); } } Utility.WorkOnText( "Hello world", new PessimisticTransformer() ); That's quite a huge difference. It's surprising that C# hasn't gotten this functionality yet. It's hard to see what the downside is for this feature -- it doesn't seem to alter semantics. While it is supported in Java, there are other restrictions. The signature has to match exactly. What happens if we add an optional parameter to the interface-method definition? public interface ITransformer { string Adjust(string value, ITransformer additional = null); } In the C# example, the class implementing the interface would have to be updated, of course, but the code at calling location remains unchanged. The functional interface's definition is the calling location, so the change would be closer to the implementation instead of more abstracted from it. public class PessimisticTransformer : ITransformer { public string Adjust(string value, ITransformer additional = null) { return value.Replace("Hello", "Goodbye cruel"); } } // Using a class Utility.WorkOnText( "Hello world", new PessimisticTransformer() ); // Using a functional interface Utility.WorkOnText( "Hello world", (s, a) => s.Replace("Hello", "Goodbye cruel") ); I would take the functional interface any day. [Java Closures] As a final note, Java 8 has finally acquired closures/lambdas [4] but there is a limitation on which functions can be passed as lambdas. It turns out that the inclusion of functional interfaces is a workaround for not having first-class functions in the language. Citing the article, "[...] you cannot pass any function as first-class to other functions, the function must be explicitly defined as lambda or using Functional Interfaces" While in C# you can assign any method with a matching signature to a lambda variable or parameter, Java requires that the method be first assigned to a variable that is "explicitly assigned as lambda" in order to use. This isn't a limitation on expressiveness but may lead to clutter. In C# I can write the following: public string Twist(string value) { return value.Reverse(); } public string Alter(this string value, Func func) { return func(value); } public string ApplyTransformations(string value) { return value.Alter(Twist).Alter(s => s.Reverse()); } This example shows how you can declare a Func to indicate that the parameter is a first-class function. I can pass the Twist function or I can pass an inline lambda, as shown in ApplyTransformations. However, in Java, I can't declare a Func: only functional interfaces. In order to replicate the C# example above in Java, I would do the following: public String twist(String value) { return new StringBuilder(value).reverse().toString(); } public String alter(String value, ITransformer transformer) { return transformer.adjust(value); } public String applyTransformations(String value) { return alter(alter(value, s -> twist(s)), s -> StringBuilder(s).reverse().toString(); } Note that the Java example cannot pass Twist directly; instead, it wraps it in a lambda so that it can be passed as a functional interface. Also, the C# example uses an extension method, which allows me to "add" methods to class string, which is not really possible in Java. Overall, though, while these things feel like deal-breakers to a programming-language snob [5] -- especially those who have a choice as to which language to use -- Java developers can rejoice that their language has finally acquired features that both increase expressiveness and reduce clutter. [6] As a bonus, as a C# developer, I find that I don't have to be so jealous after all. Though I'd still really like me some functional interfaces. -------------------------------------------------------------------------------- [1] Even if I were still a Java programmer, the champagne might still stay in the bottle because adoption of the latest runtime in the Java world is extremely slow-paced. Many projects and products require a specific, older version of the JVM and preclude updating to take advantage of newer features. The .NET world naturally has similar limitations but the problem seems to be less extreme. [1] Distraught because the features look quite interesting and useful and C# doesn't have them and pleased because (A) I am not so immature that I can't be happy for others and (B) I know that innovation in other languages is an important driver in your own language. [1] Totally kidding here. I'm not insane. Take my self-diagnosis with a grain of salt. [1] I know that lambdas and closures are not by definition the same and I'm not supposed to use the interchangeably. I'm trying to make sure that a C# developer who reads this article doesn't read "closure" (which is technically what a lambda in C# is because it's capable of "closing over" or capturing variables) and not understand that it means "lambda". [1] Like yours truly. [1] Even if most of those developers won't be able to use those features for quite some time because they work on projects or products that are reluctant to upgrade. ]]> https://www.earthli.com/news/view_article.php?id=2953 2014-03-13T21:46:59+01:00 https://earthli.com/users/marco

we took a look at how Quino seamlessly maps as much as possible to the database, while handling unmappable query components locally as efficiently as possible. [Correctness is more important than efficiency] As efficiently as possible can be a bit of a weasel... ]]>

we took a look at how Quino seamlessly maps as much as possible to the database, while handling unmappable query components locally as efficiently as possible. [Correctness is more important than efficiency] As efficiently as possible can be a bit of a weasel statement. We saw that partial application of restrictions could significantly reduce the data returned. And we saw that efficient handling of that returned data could minimize the impact on both performance and memory, keeping in mind, of course, that the primary goal is correctness. However, as we saw in the previous article, it's still entirely possible that even an optimally mapped query will result in an unacceptable memory-usage or performance penalty. In these cases, we need to be able to hint or warn the developer that something non-optimal is occurring. It would also be nice if the developer could indicate whether or not queries with such deficiencies should even be executed. [When do things slow down?] Why would this be necessary? Doesn't the developer have ultimate control over which queries are called? The developer has control over queries in business-logic code. But recall that the queries that we are using are somewhat contrived in order to keep things simple. Quino is a highly generic metadata framework: most of the queries are constructed by standard components from expressions defined in the metadata. For example, the UI may piece together a query from various sources in order to retrieve the data for a particular view. In such cases, the developer has less direct control to "repair" queries with hand-tuning. Instead, the developer has to view the application holistically and make repairs in the metadata. This is one of many reasons why Quino has local evaluation and does not simply throw an exception for partially mapped queries, as EF does. [Debugging data queries] [image]It is, in general, far better to continue working while executing a possibly sub-optimal and performance-damaging query than it is to simply crash out. Such behavior would increase the testing requirements for generated UIs considerably. Instead, the UI always works and the developer can focus on optimization and fine-tuning in the model, using tools like the Statistics Viewer, shown to the left. [image]The statistics viewer shows all commands executed in an application, with a stack trace, messages (hints/warnings/info) and the original query and mapped SQL/remote statement for each command. The statistics are available for SQL-based data drivers, but also for remoting drivers for all payload types (including JSON). The screenshot above is for the statistics viewer for Winform applications; we've also integrated statistics into web applications using "Glimpse" , a plugin architecture for displaying extra information for web-site developers. The screenshot to the right shows a preview-release version that will be released with Quino 1.11 at the end of March. [Sorting is all or nothing] One place where an application can run into efficiency problems is when the sort order for entities is too complex to map to the server. If a single restriction cannot be mapped to the database, we can map all of the others and evaluate the unmappable ones locally. What happens if a single sort cannot be mapped to the database? Can we do the same thing? Again, to avoid being too abstract, let's start with an example. var query = Session.GetQuery(); query .Where(Person.Fields.LastName, ExpressionOperator.StartsWith[1], "M") .OrderBy(Person.Fields.LastName) .OrderBy(Person.Fields.FirstName) .Join(Person.Relations.Company).WhereEqual(Company.Fields.Name, "IBM"); Assert.That(Session.GetList(query).Count, Is.Between(100, 120)); Both of these sorts can be mapped to the server so the performance and memory hit is very limited. The ORM will execute a single query and will return data for and create about 100 objects. Now, let's replace one of the mappable sorts with something unmappable: var query = Session.GetQuery(); query .Where(Person.Fields.LastName, ExpressionOperator.StartsWith[1], "M") .OrderBy(new DelegateExpression(c => c.GetObject().FirstName) .OrderBy(Person.Fields.LastName) .Join(Person.Relations.Company).WhereEqual(Company.Fields.Name, "IBM"); Assert.That(Session.GetList(query).Count, Is.Between(100, 120)); What's happening here? Instead of being able to map both sorts to the database, now only one can be mapped. Or can it? The primary sort can't be mapped, so there's obviously no point in mapping the secondary sort. Instead, all sorting must be applied locally. What if we had been able to map the primary sort but not the secondary one? Then we could have the database apply the primary sort, returning the data partially ordered. We can apply the remaining sort in memory...but that won't work, will it? If we only applied the secondary sort in memory, then the data would end up sort only by that value. It turns out that, unlike restrictions, sorting is all-or-nothing. If we can't map all sorts to the database, then we have to apply them all locally. [1] In this case, the damage is minimal because the restrictions can be mapped and guarantee that only about 100 objects are returned. Sorting 100 objects locally isn't likely to show up on the performance radar. Still, sorting is a potential performance-killer: as soon as you stray from the path of standard sorting, you run the risk of either: * Choosing a sort that is mappable but not covered by an index on the database * Choosing a sort that is unmappable and losing out on index-optimized sorting on the database In the next article, we'll discuss how we can extract slices from a result set -- using limit and offset -- and what sort of effect this can have on performance in partially mapped queries. -------------------------------------------------------------------------------- [1] The mapper also doesn't bother adding any ordering to the generated query if at least one ordering is unmappable. There's no point in wasting time on the database with a sort that will be re-applied locally. ]]> https://www.earthli.com/news/view_article.php?id=2951 2014-03-06T22:33:32+01:00 https://earthli.com/users/marco

we took a look at some of the basics of querying data with Quino while maintaining acceptable performance and memory usage. Now we'll take a look at what happens with partially-mapped queries. Before explaining what those are, we need a... ]]>

we took a look at some of the basics of querying data with Quino while maintaining acceptable performance and memory usage. Now we'll take a look at what happens with partially-mapped queries. Before explaining what those are, we need a more concrete example to work with. Here's the most-optimized query we ended up with in the previous article: var query = Session.GetQuery(); query.Join(Person.Relations.Company).WhereEqual(Company.Fields.Name, "IBM"); Assert.That(Session.GetCount(query), Is.GreaterThanEqual(140000)); With so many entries, we'll want to trim down the list a bit more before we actually create objects. Let's choose only people whose last names start with the letter "M". var query = Session.GetQuery(); query .Where(Person.Fields.LastName, ExpressionOperator.StartsWith [1], "M") .Join(Person.Relations.Company).WhereEqual(Company.Fields.Name, "IBM"); Assert.That(Session.GetCount(query), Is.Between(100, 120)); This is the kind of stuff that works just fine in other ORMs, like Entity Framework. Where Quino goes just a little farther is in being more forgiving when a query can be only partially mapped to the server. If you've used EF for anything beyond trivial queries, you've surely run into an exception that tells you that some portion of your query could not be mapped. [2] Instead of throwing an exception, Quino sends what it can to the database and uses LINQ to post-process the data sent back by the database to complete the query. [Introducing unmappable expressions] Unmappable code can easily sneak in through aspects in the metadata that define filters or sorts using local methods or delegates that do not exist on the server. Instead of building a complex case, we're going to knowingly include an unmappable expression in the query. var query = Session.GetQuery(); query .Where(new DelegateExpression [3](c => c.GetObject().LastName.StartsWith("M") [4]) .Join(Person.Relations.Company).WhereEqual(Company.Fields.Name, "IBM"); Assert.That(Session.GetCount(query), Is.Between(100, 120)); The new expression performs the same check as the previous example, but in a way that cannot be mapped to SQL. [5] With our new example, we've provoked a situation where any of the following could happen: * The ORM could throw up its hands and declare the query unmappable, pushing the responsibility for separating mappable from unmappable onto the shoulders of the developers. As noted above, this is what EF does. * The ORM could determine that the query is unmappable and evaluate everything locally, retrieving only the initial set of Person objects from the server (all several million of them, if you'll recall from the previous post). * The ORM could map part of the query to the database, retrieving the minimal set of objects necessary in order to guarantee the correct result. This is what Quino does. The strategy works in many cases, but is not without its pitfalls. What happens when we evaluate the query above? With partial mapping, we know that the restriction to "IBM" will be applied on the database. But we still have an additional restriction that must be applied locally. Instead of being able to get the count from the server without creating any objects, we're now forced to create objects in memory so that we can apply the local restrictions and only count the objects that match them all. But as you'll recall from the previous article, the number of matches for "IBM" is 140,000 objects. The garbage collector just gave you a dirty look again. [Memory bubbles] There is no way to further optimized this query because of the local evaluation, but there is a way to avoid another particularly nasty issue: memory bubbles. What is a memory bubble you might ask? It describes what happens when your application is using nMB and then is suddenly using n + 100MB because you created 140,000 objects all at once. Milliseconds later, the garbage collector is thrashing furiously to clean up all of these objects -- and all because you created them only in order to filter and count them. A few milliseconds after that, your application is back at nMB but the garbage collector's fur is ruffled and it's still trembling slightly from the shock. The way to avoid this is to stream the objects through your analyzer one at a time rather than to create them all at once. Quino uses lazily-evaluated IEnumerable sequences throughout the data driver specifically to prevent memory bubbles. [Streaming with IEnumerable sequences] Before tackling how the Quino ORM handles the Count(), let's look at how it would return the actual objects from this query. * Map the query to create a SELECT statement * At this point, it doesn't matter whether the entire query could be mapped * Create an IEnumerable sequence that represents the result of the mapped query * At this point, nothing has been executed and no objects have been returned * Wrap the sequence in another sequence that applies all of the "unhandled" parts of the query * Return that sequence as the result of executing the query * At this point, we still haven't actually executed anything on the database or created any objects Right, now we have an IEnumerable that represents the result set, but we haven't lit the fuse on it yet. How do we light the fuse? Well, the most common way to do so is to call ToList() on it. What happens then? * The IEnumerator requests an element * The query is executed against the database and returns an IDataReader * The reader requests a row and creates a Person object from that row's data * The wrapper that performs the local evaluation applies its filter(s) to this Person and yields it if it matches * If it matched the local filters, the Person is added to the list * Control returns to the IDataReader, which requests another row * Repeat until no more rows are returned from the database Since the decision to add all objects to a list occurs all the way at the very outer caller, it's the caller that's at fault for the memory bubble not the driver. [6] We'll see in the section how to avoid creating a list when none is needed. Using cursors to control evaluation If we wanted to process data but perhaps offer the user a chance to abort processing at any time, we could even get an IDataCursor from the Quino ORM so control iteration ourselves. using (var cursor = Session.CreateCursor(query)) { foreach (var obj in cursor) { // Do something with obj if (userAbortedOperation) { break; } } } [And finally, the count query] But back to evaluating the query above. The Quino ORM handles it like this: * Try to map the query to create a COUNT statement * Notice that at least one restriction could not be mapped * Create a cursor to SELECT all of the objects for the same query (shown above) and count all the objects returned by that instead So, if a count-query cannot be fully mapped to the database, the most efficient possible alternative is to execute a query that retrieves as few objects as possible (i.e. maps as much to the server as it can) and streams those objects to count them locally. Tune in next time for a look at how to exert more control with limit and offset and how those work together with partial mapping. -------------------------------------------------------------------------------- [1] These exceptions are not always obvious, either. Often, the exception complains about not being able to call a static method on a non-static object or class, which occurs when mapping the Expression object to SQL. EF's error messages have in general gotten much better but the mapping failures can be quite confusing. [1] If we were worried that the last names in our database might not necessarily be capitalized, we would use the ExpressionOperator.StartsWithCI to perform the check in a case-insensitive manner instead. [1] A DelegateExpression simply wraps the lambda given in the constructor in a Quino expression object. The parameter c is an IExpressionContext that provides the target object, which is in this case a Person. [1] I'm going to go ahead and assume that my database schema doesn't allow nulls for the LastName field. [1] If Quino had a LINQ-to-SQL provider, there's a chance that more of these delegates could be mapped, but we don't have one...and they can't. [1] Did we still create 140,000 objects? Yes we did, but not all at once. Now, there are probably situations where it is better to create several objects rather than streaming them individually, but I'm confident that keeping this as the default is the right choice. If you find that your particular situation warrants different behavior, feel free to use Session.CreateCursor() to control evaluation yourself and create the right-sized batches of objects to count. The ChangeAndSave() extension method does exactly that to load objects in batches (size adjustable by an optional parameter) rather than one by one. ]]> https://www.earthli.com/news/view_article.php?id=2965 2014-02-24T23:01:09+01:00 https://earthli.com/users/marco

. There users suggested various solutions and the original poster became ever more adamant -- "Suffice it to say that we believe we need default variable setting as we've proposed here" -- until a LESS developer waded in to state that it would be "a pointless feature in less", which seemed harsh until an example showed that he was quite right. The clue is further down in one of the answers: "If users define overrides after then it works as if it had a default on it. [T]hat's because even in the imported file it will take the last definition in the same way as css, even if defined after usage. (Emphasis added.)" It was at this point that the lightbulb went on for me. I was thinking like a programmer where a file is processed top-down and variable values can vary depending on location in the source text. That the output of the following C# code is 12 should amaze no one. var a = 1; Console.Write(a); a = 2; Console.Write(a); a = 3; In fact, we would totally expect our IDE to indicate that the value in the final assignment is never used and can be removed. Using LESS variable semantics, though, where variables are global in scope [3] and assignment are treated as they are in CSS, we would get 33 as output. Why? Because the value of the variable a has the value 3 because that's the last value assigned to it. That is, LESS has a cascading approach to variable assignment. This is exactly as the developer from LESS said: stop fighting it and just let LESS do what it does best. Do you want default values? Define the defaults first, then define your override values. The overridden value will be used even when used for setting the value of another default value that you didn't even override. Now let's go fix our stylesheet to use these terse semantics of LESS. Here's a first cut at a setup that feels pretty right. I put the files in the order that you would read them so that you can see the overridden values and everything makes sense again. [4] ------------------------ | theme-variables.less | ------------------------ @body_color: white; @title_color: darken(@body_color, 25%); @border_color: @title_color; ------------------------ ---------------- | crimson.less | ---------------- @import "theme-variables"; @body_color: #800; @import "theme-base"; ---------------- ------------------- | theme-base.less | ------------------- body { background-color: @body_color; } h2 { color: @title_color; border: 1px solid @border_color; } ------------------- You can see in the example above that the required variables are all declared, then overridden and then used. From what we learned above, we know that the value of @title_color in the file theme-variables.less will use a value of #800 for @body_color because that was the last value it was assigned. We can do better though. The example above hasn't quite embraced the power of LESS fully. Let's try again. ------------------- | theme-base.less | ------------------- @body_color: white; @title_color: darken(@body_color, 25%); @border_color: @title_color; body { background-color: @body_color; } h2 { color: @title_color; border: 1px solid @border_color; } ------------------- ---------------- | crimson.less | ---------------- @import "theme-base"; @body_color: #800; ---------------- Boom! That's all you have to do. Set up everything in your base stylesheet file. Define all variables and define them in terms of each other in as convoluted a manner as you like. The final value of each value is determined before any CSS is generated. This final version also has the added advantage that a syntax-checking IDE like JetBrains WebStorm or PHPStorm will be able to provide perfect assistance and validity checking. That wasn't true at all for any of the previous versions, where variable declarations were in different files. Although I was seriously considering moving away from LESS and over to SASS -- because at least they didn't leave out such a basic feature, as I had thought crossly to myself -- I'm quite happy to have learned this lesson and am more happy with LESS than ever. -------------------------------------------------------------------------------- [1] For those of you who already know how to fix this, stop smirking. I'm writing this post because it wasn't intuitive for me -- although now I see the utter elegance of it. [1] I'd also seen the same concept in NAnt property tasks where you can use the now-deprecated overwrite="false" directive. For the curious, now you're supposed to use unless="${property::exists('property-name')}" instead, which is just hideous. [1] There are exceptions, but "variables are global in LESS is a good rule of thumb". One example is that if a parameter for a mixin has the same name as a globally assigned variable, the value within that mixin is taken from the parameter rather than the global. [1] Seriously, LESS experts, stop smirking. I'm taking a long time to get there because a programmer's intuitive understanding of how variables work is a hard habit to break. Almost there. ]]> https://www.earthli.com/news/view_article.php?id=2948 2014-02-09T23:08:59+01:00 https://earthli.com/users/marco

contains a lot of interesting information, related to only to parsing, but also to runtime and framework design. Bright is well-known as the designer of the D programming language, so he's definitely worth a read. I thought he jumped... ]]>

contains a lot of interesting information, related to only to parsing, but also to runtime and framework design. Bright is well-known as the designer of the D programming language, so he's definitely worth a read. I thought he jumped back and forth between topics a bit, so I summarized the contents for myself below: [Parsing] Bright identifies Minimizing keystrokes, easy parsing and minimizing the number of keywords as false gods. Do not waste any time trying to satisfy these requirements; instead, let them flow naturally from a good design. Your language should consist of productions that have only a single non-terminal on the left-hand side. That is, strive to make your language context-free. [1] The implication is that you're actually going to define the grammar rather than just winging it. This means that you can can use a parser generator even though Bright says not to "bother wasting time with lexer or parser generators and other so-called 'compiler compilers.'" I instead agree with the article "Advice on writing a programming language" by Ted Kaminski' , which advises providing a grammar that can be used with parser generators because "many of those people eager to contribute either get stuck trying and failing to build a parser or trying and failing to learn to use the daunting internals of your compiler". You can either make it easy for people to build compilers for your language or you can maintain a very friendly API for your own compiler. If you choose the API route, it might force you to be more disciplined, but it might also cause you no end of backwards-compatibility headaches as your compiler quickly evolves. Not only that, but you'd then have to make that API available for any number of languages and any number of platforms. If you take the route of publishing the BNF, that may also not not be enough. This because it can still be daunting to convert a BNF to something that your compiler-generator can use, especially for non-trivial languages. Providing a grammar for a widely supported parser-generator like "ANTLR" [2] will give those willing to build tools for your language a good jump-start. "Use an LR parser generator. It’ll keep your language parsable, and make it easier to change early on. When your language becomes popular enough that you have the problem of parsing errors not being friendly enough for your users, celebrate that fact and hand-roll only then. "And then reap the benefit of all the tooling an easily parsed language gets, since you know you kept it LR(1)." [Error-handling] Introduce redundancy into the language definition (e.g. semicolons as line-terminators in addition to whitespace/newlines) in order to make error-message generation much easier and much more likely to produce friendly output. Compilers can handle error messages in different ways: Bail out on the first error This is a good fallback, but it saves the developer a lot of work if you identify all of the root errors in source -- that is, errors that are not a consequence of another error. Collect multiple errors In order to continue parsing/compiling after an error, the machine can take one of two approaches: * "Guess what the programmer intended, repair the syntax trees, and continue." (Bright) Bad guesses lead to spurious and inscrutable error messages which lead to developers no longer trusting their compilers. Avoid this approach as it is very difficult to get right. * Take the approach that Bright did with the D compiler: consider any part of the code that has an error as "poisoned". He likens it to the way that "floating-point NaNs are handled. Any operation with a NaN operand silently results in a NaN." With this approach, "the compiler is able to detect multiple errors as long as the errors are in sections of code with no dependency between them", which yields only high-quality and relevant error messages for the developer. [Stand on the shoulders of giants] Do not re-invent the syntax for everything in your language. Instead, as Bright says, "[s]ave the divergence for features not generally seen before, which also signals the user that this is new." [The runtime] A language definition is nothing without a runtime. Bright recommends "taking the common sense approach and using an existing back end, such as the JVM, CLR, gcc, or LLVM. (Of course, I can always set you up with the glorious Digital Mars back end!)" If you can avoid writing your own back-end, you should definitely do so. Similar to the approach recommended for parsing the language: start with a stock runtime and migrate to something custom if the needs of your project warrant it (they almost certainly won't). This is the approach taken by any number of other popular languages, like Scala. [The framework] And then there's the library/framework that accompanies the language and, arguably, helps to define it for people. Complaints about a language are often complaints about the standard runtime library/framework for the language. Developers quickly associate them and treat them as one entity. Bright's focus is on very low-level runtimes (such as the one for his language, D) and thus his advice focuses on fast I/O, fast and efficient memory allocation/de-allocation and robust/fast transcendental functions [3]. However, he also offers the following excellent rule of thumb for any framework: "My general rule is if the explanation for what the function does is more lines than the implementation code, then the function is likely trivia and should be booted out." -------------------------------------------------------------------------------- [1] See "Example of why C++ is NOT a context free grammar?" by Kaz Kylheku and "Context-free grammar" for more information. [1] These are functions that cannot be composed of other functions in the framework, what I would call "core" or "root" functions. These are the functions that the developer would find it either impossible or incredibly difficult to replicate efficiently in the language itself. [1] As of publication time, the current stable release of ANTLR for C# available via NuGet -- and used by most other packages -- is version 3.5.1. However, the official home page is now "ANTLR" rather than "ANTLR3" . The NuGet packages for version 4 are in pre-release, but it's very nice to see some progress being made after years of relatively minor upgrades. In particular, the new version of the IDE "ANTLRWorks" looks much nicer and seems to have been based on the JETBrains IDE framework. I'm definitely looking forward to checking it out in more detail. ]]> https://www.earthli.com/news/view_article.php?id=2950 2014-02-07T09:57:07+01:00 https://earthli.com/users/marco

and is cross-published here. ------------------------------------------------------------------------ One of the most-used components of Quino is the ORM. An ORM is an Object-Relational Mapper, which accepts queries and returns data. * Applications formulate queries in Quino using application metadata ... ]]>

and is cross-published here. -------------------------------------------------------------------------------- One of the most-used components of Quino is the ORM. An ORM is an Object-Relational Mapper, which accepts queries and returns data. * Applications formulate queries in Quino using application metadata * The ORM maps this query to the query language of the target database * The ORM transforms the results returned by the database to objects (the classes for which were also generated from application metadata). This all sounds a bit abstract, so let's start with a concrete example. Let's say that we have millions of records in an employee database. We'd like to get some information about that data using our ORM. With millions of records, we have to be a bit careful about how that data is retrieved, but let's continue with concrete examples. [Attempt #1: Get your data and refine it locally] The following example returns the correct information, but does not satisfy performance or scalability requirements. [1] var people = Session.GetList().Where(p => p.Company.Name == "IBM"); Assert.That(people.Count(), Is.GreaterThanEqual(140000)); What's wrong with the statement above? Since the call to Where occurs after the call to GetList(), the restriction cannot possibly have been passed on to the ORM. The first line of code doesn't actually execute anything. It's in the call to Count() that the ORM and LINQ are called into action. Here's what happens, though: * For each row in the Person table, create a Person object * For each person object, create a corresponding Company object * Count all people where the Name of the person's company is equal to "IBM". The code above benefits from almost no optimization, instantiating a tremendous number of objects in order to yield a scalar result. The only side-effect that can be considered an optimization is that most of the related Company objects will be retrieved from cache rather than from the database. So that's a plus. Still, the garbage collector is going to be running pretty hot and the database is going to see far more queries than necessary. [3] [Attempt #2: Refine results on the database] Let's try again, using Quino's fluent querying API. [4] The Quino ORM can map much of this API to SQL. Anything that is mapped to the database is not performed locally and is, by definition, more efficient. [5] var people = Session.GetList(); people.Query.Join(Person.Relations.Company).WhereEqual(Company.Fields.Name, "IBM"); [6] Assert.That(people.Count, Is.GreaterThanEqual(140000)); First, we get a list of people from the Session. As of the first line, we haven't actually gotten any data into memory yet -- we've only created a container for results of a certain type (Person in this case). The default query for the list we created is to retrieve everything without restriction, as we saw in the first example. In this example, though, we restrict the Query to only the people that work for a company called "IBM". At this point, we still haven't called the database. The final line is the first point at which data is requested, so that's where the database is called. We ask the list for the number of entries that match it and it returns an impressive number of employees. At this point, things look pretty good. In older versions of Quino, this code would already have been sufficiently optimized. It results in a single call to the database that returns a single scalar value with everything calculated on the database. Perfect. [Attempt #3: Avoid creating objects at all] However, since "v1.6.0" of Quino [7], the call to the property IDataList.Count has automatically populated the list with all matching objects as well. We made this change because the following code pattern was pretty common: var list = Session.GetList(); // Adjust query here if (list.Count > 0) { // do something with all of the objects here } That kind of code resulted in not one, but two calls to the database, which was killing performance, especially in high-latency environments. That means, however, that the previous example is still going to pull 14,000 objects into memory, all just to count them and add them to a list that we're going to ignore. The garbage collector isn't a white-hot glowing mess anymore, but it's still throwing you a "look of disapproval" . Since we know that we don't want the objects in this case, we can get the old behavior back by making the following adjustment. var people = Session.GetList(); people.Query.Join(Person.Relations.Company).WhereEqual(Company.Fields.Name, "IBM"); Assert.That(Session.GetCount(people.Query), Is.GreaterThanEqual(140000)); It would be even clearer to just forget about creating a list at all and work only with the query instead. var query = Session.GetQuery(); query.Join(Person.Relations.Company).WhereEqual(Company.Fields.Name, "IBM"); Assert.That(Session.GetCount(query), Is.GreaterThanEqual(140000)); Now that's a maximally efficient request for a number of people in Quino 1.10 as well. Tune in next time for a look at what happens when a query can only be partially mapped to the database. -------------------------------------------------------------------------------- [1] I suppose it depends on what those requirements are, but if you think your application's performance requirements are so loose that it's OK to create millions of objects in memory just in order to count them, then you're probably not in the target audience for this article. [1] Instead of using a code-first, Quino uses a "metadata-first" approach, so the Person class used here is generated from the application metadata rather than written by the developer, as in other frameworks. There are different strategies for retrieving associated data. Quino does not yet support retrieving anything other than root objects. That is, the associated Company object is not retrieved in the same query as the Person object. In the example in question, the first indication that the ORM has that a Company is required is when the lambda retrieves them individually. Even if the original query had somehow indicated that the Company objects were also desired (e.g. using something like Include(Person.Relations.Company) as you would in EF), the most optimal mapping strategy is still not clear. Should the mapper join the company table and retrieve that highly redundant data with each person? Or should it execute a single query for all companies and prime a cache with those? The right answer depends on the latency and bandwidth between the ORM and the database as well as myriad other conditions. When dealing with a lot of data, it's not hard to find examples where the default behavior of even a clever ORM isn't maximally efficient -- or even very efficient at all. As we already noted, though, the example in question does everything in memory. If we reasonably assume that the people belong to a relatively small number of companies -- say qc -- then the millions of calls to retrieve companies associated with people will result in a lot of cache hits and generate "only" qc + 1 queries. [1] Quino does not have LINQ to SQL support. I'm not even going to write "yet" at the end of that sentence because it's not at all clear that we're ever going to have it. Popular demand might convince us otherwise, but for now we're quite happy with our API (and soon-to-be-revealed query language QQL). [1] That's an assumption I'm going to make for which counterexamples certainly exist, but none of which apply to the simple examples we'll address in this article. [1] The Person.Relations and Person.Fields static fields are generated with the Person class. These correspond to the application metadata and change when the metadata changes. Developers are encouraged to use these generated constants so that even metadata-based queries can be validated by the compiler. [1] That was in almost three years ago, in June of 2011. ]]> https://www.earthli.com/news/view_article.php?id=2943 2014-02-01T17:09:41+01:00 https://earthli.com/users/marco

. We took a look at two contenders: Dart and TypeScript. At the time, Dart was weaker for the following reasons: * It had not yet been released * It had little ... ]]>

. We took a look at two contenders: Dart and TypeScript. At the time, Dart was weaker for the following reasons: * It had not yet been released * It had little to no tool support * Integration with existing JS libraries was somewhat laborious Though TypeScript has its weaknesses (it has technically not yet hit a 1.0 release), we eventually decided to go in that direction. The tool support in Visual Studio and ReSharper are both improving steadily and have gotten quite good. We've had quite positive results in one larger project. Even with Dart in our wake, I am still curious to see how people are using it. I was surprised by the claims in the article "Why Dart should learn JSON while it’s still young" by Max Horstmann . Since Dart is not directly compatible with JavaScript, as TypeScript is, neither can a given JSON-formatted string be implicitly interpreted. Instead, you can import it using a library function. This is not really a problem, though one wonders if there are performance penalties for Dart that are not present in JavaScript/TypeScript. Where the problem arises is in exporting JSON, which does not happen automagically. In non--client-side languages like C#, NewtonSoft's JSON.Net library can serialize pretty much anything using reflection. JSON isn't baked into the language, but that isn't too surprising. However, in Dart, positioned as a contender for taking over from JavaScript as the client-side language of choice, the solution recommended even by Dart language gurus is to implement toJson() on all objects that you want to export. Either that, or use a probably non-optimized external library to serialize your object to JSON (likely using introspection, as JSON.Net does). I agree with the author of the blog that this is a red flag for using Dart in production projects. It's strange that Dart doesn't produce JSON without relying on external libraries. And the recommended library is, as of this writing, of pre-production/alpha quality -- the version number is 0.1.0 and the TODO list includes a bullet point that exhorts the author to "Write tests!". So I'm still waiting to see what becomes of Dart, but the balkiness of the current solution for generating JSON not only makes it a bit of tough fit for many current web applications, but also makes us urge caution despite its having recently been released (1.0 came out in November 2013). ]]> https://www.earthli.com/news/view_article.php?id=2942 2014-02-01T12:38:27+01:00 https://earthli.com/users/marco

. (The article is a treasure trove of other gems, no pun intended.) The example below shows how that might work. class RandomSubclass < [Array, Hash, String, Fixnum, Float, TrueClass].sample end RandomSubclass.superclass # could output one of 6 different classes. The language allows you to call methods from the "extends" clause. The example above creates an array of class names, then calls the sample method on them to yield a base class. The actual base class is not only unknown at compile time, it is also unpredictable at runtime. The example above is contrived and makes the feature seem like it's only for the reckless. It's clear that serious software would have to forbid or strictly limit the use of such a feature, but I can see where it would be useful. For example, you may want to change your base class depending on deployment parameters. If you're deploying to a testing or staging environment, you'll use a base class that includes more logging, profiling and debugging code. For production, you switch to a base class that's optimized. If the class interface remains the same, then using this feature wouldn't be as dangerous as it initially appeared. Still, ensuring quality and enforcing architecture in software written in such a language would require a strict development process and discipline and vigilance from all involved. ]]> https://www.earthli.com/news/view_article.php?id=2909 2014-01-05T11:46:53+01:00 https://earthli.com/users/marco

... ]]>

. It argues that the new stuff in iOS 7 is compelling enough to make developers consider dropping support for all older operating systems. And this for pragmatic reasons, such as having far less of your own code to support and correspondingly making the product cost less to support. It's best to check your actual target market, but Apple users tend to upgrade very quickly and reliably, so an iOS 7-only strategy is a good option. Among the improvements that Apple has brought in the recent past are blocks (lambdas), GCD (asynchronous execution management) and ARC (mostly automated memory management), all introduced in iOS 4 and OS X 10.6 Snow Leopard. OS X 10.9 Mavericks and iOS 7 introduced a slew of common UI improvements (e.g. AutoLayout and HTML strings for labels). [1] To find the videos listed below, browse to "WWDC 2013 Development Videos" . For the web, Apple has improved developer tools and support in Safari considerably. There are two pretty good videos demonstrating a lot of these improvements: #601: Getting to Know Web Inspector This video shows a lot of improvements to Safari 7 debugging, in the form of a much more fluid and intuitive Web Inspector and the ability to save changes made there directly back to local sources. #603: Getting the Most Out of Web Inspector This video shows how to use the performance monitoring and analysis tools in Safari 7. The demonstration of how to optimize rendering and compositing layers was really interesting. For non-web development, Apple has been steadily introducing libraries to provide support for common application tasks, the most interesting of which are related to UI APIs like Core Image, Core Video, Core Animation, etc. Building on top of these, Apple presents the Sprite Kit -- for building 2D animated user interfaces and games -- and the Scene Kit -- for building 3D animated user interfaces and games. There are some good videos demonstrating these APIs as well. #500: What’s New in Scene Kit An excellent presentation content-wise; the heavily accented English is sometimes a bit difficult to follow, but the material is top-notch. #502: Introduction to Sprite Kit This is a good introduction to nodes, textures, actions, physics and the pretty nice game engine that Apple delivers for 2D games. #503: Designing Games with Sprite Kit The first half is coverage of tools and assets management along with more advanced techniques. The second half is with game designers Graeme Devine [2] and Spencer Lindsay, who designed the full-fledged online multi-player game Adventure to showcase the Sprite Kit. -------------------------------------------------------------------------------- [1] Disclaimer: I work with C# for Windows and HTML5 applications of all stripes. I don't actually work with any of these technologies that I listed above. The stuff looks fascinating, though and, as a framework developer, I'm impressed by the apparent cohesiveness of their APIs. Take recommendations with a grain of salt; it could very well be that things are a good deal less rosy when you actually have to work with these technologies. [1] Formerly of Trilobyte and then id Software, now at Apple. ]]> https://www.earthli.com/news/view_article.php?id=2904 2013-12-29T23:09:53+01:00 https://earthli.com/users/marco

. He is such a funny writer that this article, against convention, will consist mostly of citations rather than an even mix of citations and paraphrasing that I naturally consider to be much more lucid and pithy. I... ]]>

. He is such a funny writer that this article, against convention, will consist mostly of citations rather than an even mix of citations and paraphrasing that I naturally consider to be much more lucid and pithy. I quote at length to do the material justice, for documentation and to ensure that you all download the PDFs to see if there is more where that came from (there is). All emphases have been added. "Mobile Computing Research Is a Hornet’s Nest of Deception and Chicanery" by James Mickens (PDF) On the delusions of the mobile-computing world: "Mobile computing researchers are a special kind of menace. They don’t smuggle rockets to Hezbollah, or clone baby seals and then make them work in sweatshops for pennies a day. That’s not the problem with mobile computing people. The problem with mobile computing people is that they have no shame. They write research papers with titles like “Crowdsourced Geolocation-based Energy Profiling for Mobile Devices,” as if the most urgent deficiency of smartphones is an insufficient composition of buzzwords." On browsing web pages: "When I use a mobile browser to load a web page, I literally have no expectation that anything will ever happen. A successful page load is so unlikely, so unimaginable, that mobile browsers effectively exist outside of causality—the browser is completely divorced from all action verbs, and can only be associated with sad, falling-tone sentences like “I had to give up after twenty seconds.” " On the fragility of touchscreens: "Note that, when I say that you will “drop” your touchscreen, I do not mean “drop” in the layperson sense of “to release from a non-trivial height onto a hard surface.” I mean “drop” in the sense of “to place your touchscreen on any surface that isn’t composed of angel feathers and the dreams of earnest schoolchildren.” Phones and tablets apparently require Planck-scale mechanical alignments, such that merely looking at the touchscreen introduces fundamental, quantum dynamical changes in the touchscreen’s dilithium crystals. Thus, if you place your touchscreen on anything, ever, you have made a severe and irreversible life mistake." On the sheer touchiness of touchscreens: "On your touchscreen, your swipes will become pinches, and your pinches will become scrolls, and each one of your scrolls will become a complex thing never before seen on this earth, a leviathan meta-touch event of such breadth and complexity that your phone can only respond like Carrie White at the prom. So, your phone just starts doing stuff, all the stuff that it knows how to do, and it’s just going nuts, and your apps are closing and opening and talking to the cloud and configuring themselves in unnatural ways, and your phone starts vibrating and rumbling with its little rumble pack, and it will gently sing like a tiny hummingbird of hate, and you’ll look at the touchscreen, and you’ll see that things are happening, my god, there are so many happenings, and you’ll try to flip the phone over and take out the battery, because now you just want to kill it and move to Kansas and start over, [...]" On the uselessness of most mobile computing: "When you purchase a mobile device, you are basically saying, “I endorse the operational inefficiency of the modern bourgeoisie lifestyle, even though I could find a rock and tie a coat hanger around it and have a better chance of having a phone conversation that doesn’t sound like two monsters arguing about German poetry.”" "The Slow Winter" by James Mickens (PDF) On flying in the early 21st century: "The point is that flying in airplanes used to be fun, but now it resembles a dystopian bin-packing problem in which humans, carry-on luggage, and five dollar peanut bags compete for real estate while crying children materialize from the ether and make obscure demands in unintelligible, Wookie-like languages while you fantasize about who you won’t be helping when the oxygen masks descend." On how awesome it was being a hardware architect before things got all quantum and messy: "Of course, pride precedes the fall, and at some point, you realize that to implement aggressive out-of-order execution, you need to fit more transistors into the same die size, but then a material science guy pops out of a birthday cake and says YEAH WE CAN DO THAT, and by now, you’re touring with Aerosmith and throwing Matisse paintings from hotel room windows, because when you order two Matisse paintings from room service and you get three, that equation is going to be balanced. It all goes so well, and the party keeps getting better. When you retire in 2003, your face is wrinkled from all of the smiles, and even though you’ve been sued by several pedestrians who suddenly acquired rare paintings as hats, you go out on top, the master of your domain. " On quantum-level effects in modern processors: "They randomly switched states; they leaked voltage; they fell prey to the seductive whims of cosmic rays that, unlike the cosmic rays in comic books, did not turn you into a superhero, but instead made your transistors unreliable and shiftless, like a surly teenager who is told to clean his room and who will occasionally just spray his bed with Lysol and declare victory." On scaling in cores when processor speed and more transistors became too messy: "John did what any reasonable person would do: he cloaked himself in a wall of denial and acted like nothing had happened. “Making processors faster is increasingly difficult,” John thought, “but maybe people won’t notice if I give them more processors.” This, of course, was a variant of the notorious Zubotov Gambit, named after the Soviet-era car manufacturer who abandoned its attempts to make its cars not explode, and instead offered customers two Zubotovs for the price of one [...]" On the main purpose that people have for their computers: "Lay people use their computers for precisely ten things, none of which involve massive computational parallelism, and seven of which involve procuring a vast menagerie of pornographic data and then curating that data using a variety of fairly obvious management techniques, like the creation of a folder called “Work Stuff,” which contains an inner folder called “More Work Stuff,” where “More Work Stuff” contains a series of ostensible documentaries that describe the economic interactions between people who don’t have enough money to pay for pizza and people who aren’t too bothered by that fact. " A summary of the state of the world of hardware design and development: "[...] you brought the fire down from Olympus, and the mortals do with it what they will. But now, all the easy giants were dead, and John was left to fight the ghosts that Schrödinger had left behind." "The Night Watch" by James Mickens (PDF) What it's like to be a systems (low-level) programmer: "A systems programmer will know what to do when society breaks down, because the systems programmer already lives in a world without law." On why people still use C++ (or a response to the snotty question of: "why don't you just use high-level language X instead?") "Why not use a modern language with garbage collection and functional programming and free massages after lunch? Here’s the answer: Pointers are real. They’re what the hardware understands. Somebody has to deal with them. You can’t just place a LISP book on top of an x86 chip and hope that the hardware learns about lambda calculus by osmosis. [...] Pointers are like [...] real, living things that must be dealt with so that polite society can exist. Make no mistake, I don’t want to write systems software in a language like C++. [...] When it’s 3 A.M., and you’ve been debugging for 12 hours, and you encounter a virtual static friend protected volatile templated function pointer, you want to [...] find the people who wrote the C++ standard and bring ruin to the things that they love." On being thankful for systems programmers: "That being said, if you find yourself drinking a martini and writing programs in garbage-collected, object-oriented Esperanto, be aware that the only reason that the Esperanto runtime works is because there are systems people who have exchanged any hope of losing their virginity for the exciting opportunity to think about hex numbers and their relationships with the operating system, the hardware, and ancient blood rituals that Bjarne Stroustrup performed at Stonehenge." On how difficult it is to work in extremely fragile territory (rather than a safe runtime): "Indeed, I would [have...checked the log files for errors] if I hadn’t broken every component that a logging system needs to log data. I have a network file system, and I have broken the network, and I have broken the file system, and my machines crash when I make eye contact with them. I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS." A backhanded swipe at the utter uselessness of many UI concerns: "I’m glad that people are working on new kinds of bouncing icons because they believe that humanity has solved cancer and homelessness and now lives in a consequence-free world of immersive sprites." ]]> https://www.earthli.com/news/view_article.php?id=2890 2013-11-07T20:56:06+01:00 https://earthli.com/users/marco

, you can learn to program in various languages. It starts off very slowly and is targeted at non-technical users. That's their claim anyway -- the material in the courses I looked at ramps up pretty quickly. Anyway, the interesting thing I saw was in their introductory test. It struck... ]]>

, you can learn to program in various languages. It starts off very slowly and is targeted at non-technical users. That's their claim anyway -- the material in the courses I looked at ramps up pretty quickly. Anyway, the interesting thing I saw was in their introductory test. It struck me as a subtle way to get you to enter your email address. I'd just recently discussed this on a project I'm working on: how can we make it fun for the user to enter personal information? The goal is not to sell that information (not yet anyway, but who knows what the future holds), but to be able to enhance -- nay, personalize -- the service. Personalizing has a bad reputation but can be very beneficial. For example, if you're using a site for free and you're going to see offers and advertisements anyway, isn't it better to enter a bit of data that will increase the likelihood that offers and ads are interesting? Each person can -- and should -- decide for the themselves what to make public, but the answer isn't always necessarily no. [How Codecademy gets your email] [image] Here they teach you how to use the "length" method by measuring your email address. Sneaky. I like it. [image] Even if you don't given them an address, they re-prompt you to enter your email, but it doesn't come across as pushy because you're taking a test. I thought that this was pretty subtle. Because of the context, people who would ordinarily be sensitive to giving up their email might not even notice. Why? Because they want to answer the question *correctly*. They don't want the site to judge them for having entered something wrong, so they do as they're told. Is Codecademy collecting emails this way? I have no way to be sure, but they'd be silly not to. ]]> https://www.earthli.com/news/view_article.php?id=2859 2013-11-03T11:17:36+01:00 https://earthli.com/users/marco

and is cross-published here. ------------------------------------------------------------------------ "The following article outlines a solution to what may end up being a temporary problem. The conditions are very specific: no server-side logic; HTTP authentication; AppCache as it is implemented by" ... ]]>

and is cross-published here. -------------------------------------------------------------------------------- "The following article outlines a solution to what may end up being a temporary problem. The conditions are very specific: no server-side logic; HTTP authentication; AppCache as it is implemented by the target platforms -- Safari Mobile and Google Chrome -- in late 2012/early 2013. The solution is not perfect but it's workable. We're sharing it here in the hope that it can help someone else or serve as a base for a better solution." [The HTML5 AppCache] The application cache is a relatively new feature that is, * Supported by all modern browsers * Uses a manifest file that indicates which files to cache * Browser checks manifest for changes * If there are changes, all files are refreshed * External links work when online * When offline, the application works with the local cache * External links to non-cached content are redirected to fallback links [AppCache Limitations] Web applications can use the HTML5 application-cache to store local content, but different browsers apply different restrictions to the amount of space allocated per domain. * Safari Mobile is limited to 50MB per domain. This means that the restriction will generally apply to all content packages downloaded from the same server/domain * Google Chrome is limited as well, but the actual limit is a bit of a moving target [Optimizing the HTML5 AppCache for Authenticated Content] In particular, the Safari Mobile browser cannot update the application cache for files for which it must obtain authentication. * Some requests do not trigger authentication * Manifest file * Home-screen icons * A lost connection or timeout can invalidate the authentication token * Version checks are not reliable * Open pages/running apps do not check for status updates * Home-screen apps don’t reliably check on startup * This can lead to out-of-date or missing content [Checking for and presenting updates to the user] The graphic below illustrates the mechanism by which a content package in a web application can manage content updates and present them to the user. * When online, the software regularly checks whether an update for the package is available * The user can determine whether to install an update * When an update has been found, the software stops checking for updates until the user has applied the latest update * If the user delays the update, the user interface displays an “update” button * The software will automatically start checking for updates whenever it detects that it is online * There is no way for a user to ignore updates * When the user proceeds with an update, the latest version is retrieved at that time, ensuring that the user has the latest version [image] [Solving the problems with authenticated data and the AppCache] In order to address the problems described above, the UA products use a separate version file to check for updates independent of the browser’s application-cache mechanism and to trigger this update only when authentication has been reestablished. [image] * The cache.version.txt file is publicly available but is very small and includes only a unique version number that is also included in the cache.manifest file (both of which are generated by a deployment script). * The software compares this version number against the last known good version number. If it differs, it knows that the server has been updated with new content for this package * Before the software can kick off the HTML5 AppCache update process, it must ensure that the user is authenticated and authorized to retrieve the update package (because most browsers will simply fail silently if this is not the case). * The software pulls the force.password.txt file from the private zone with an explicit request. The browser will ask the user to authenticate, if necessary. This file is also very small to avoid needlessly downloading a large amount of data simply to force re-authentication. * Once the user has authenticated, the software lets the automated HTML5 AppCache update take over, retrieving first the cache.manifest file and then updating files as needed. The user is notified that this download is taking place asynchronously. * The software receives a notification from the browser that the update is complete and can record the version number and then notify the user that the update has been applied and is ready to use. This approach worked relatively well for us, although we continue to refine it based on feedback and experience. ]]> https://www.earthli.com/news/view_article.php?id=2288 2013-10-21T22:56:04+02:00 https://earthli.com/users/marco

, which includes Entity Framework 6 and introduces "a lot of new features" . It reminded me of the following query that EF generated for me, way, way back when it was still version 3.5. Here's hoping that they've taken care of this problem since then.... ]]>

, which includes Entity Framework 6 and introduces "a lot of new features" . It reminded me of the following query that EF generated for me, way, way back when it was still version 3.5. Here's hoping that they've taken care of this problem since then. So, the other day EF (v3.5) seemed to be taking quite a while to execute a query on SQL Server. This was a pretty central query and involved a few joins and restrictions, but wasn't anything too wild. All of the restrictions and joins were on numeric fields backed by indexes. In these cases, it's always best to just fire up the profiler and see what kind of SQL is being generated by EF. It was a pretty scary thing (I've lost it unfortunately), but I did manage to take a screenshot of the query plan, shown below. [image] It doesn't look too bad until you notice that the inset on the bottom right (the black smudgy thing) is a representation of the entire query ... and that it just kept going on down the page. ]]> https://www.earthli.com/news/view_article.php?id=2872 2013-07-15T01:38:50+02:00 https://earthli.com/users/marco

, taught me something I didn't know: there's a file you can use to ignore files in your local Git repository without changing anyone else's repository. Just to recap, here are the ways to ignore a file: * Global .gitignore: you can designate basic exclusion ... ]]>

, taught me something I didn't know: there's a file you can use to ignore files in your local Git repository without changing anyone else's repository. Just to recap, here are the ways to ignore a file: * Global .gitignore: you can designate basic exclusion directives that apply to all repositories on your system. This file is not committed to any repository or shared with others. Execute git config --global core.excludesfile ~/.gitignore_global to set the file to ~/.gitignore_global (for example). See the "linked article" for sample directives. * Per-repository global exclusions: add directives to the .git/info/exclude file in any repository. These directives are combined with any system-global directives to form the base exclusions for that repository. This file is not committed with the repository. This is the one I'd never heard of before. * .gitignore: add a file with this name to any directory. The directives in that file are merged with those from the parent directory to define the patterns that are excluded in that directory and all child directories. This is definitely the most common way to exclude files. * Exclude versioned files: and, finally, if your repository has files that are changed but not committed (e.g. configuration files), you can ignore future changes to those files with a call to git update-index --assume-unchanged path/to/file.txt. While this can be useful for legacy projects, it's best to structure new projects so developers don't have to rely on easily forgotten tricks like this. ]]> https://www.earthli.com/news/view_article.php?id=2870 2013-07-14T23:17:53+02:00 https://earthli.com/users/marco

. Recently, I read about some length units I'd never heard of in the article "CSS viewport units: vw, vh, vmin and vmax" by Chris Mills . * 1vw: 1% of viewport width * 1vh: 1% of viewport ... ]]>

. Recently, I read about some length units I'd never heard of in the article "CSS viewport units: vw, vh, vmin and vmax" by Chris Mills . * 1vw: 1% of viewport width * 1vh: 1% of viewport height * 1vmin: 1vw or 1vh, whatever is smallest * 1vmax: 1vw or 1vh, whatever is largest These should be eminently useful for responsive designs. While there is wide support for these new units, that support is only available in the absolute latest versions of browsers. See the article for a good example of how these can be used. While the ones covered in the article are actually new, there are others that have existed for a while but that I've never had occasion to use. The "Font-relative lengths: the ‘em’, ‘ex’, ‘ch’, ‘rem’ units" section lists the following units: * em: This one is well-known: 1em is equal to the "computed value of the 'font-size' property of the element on which it is used." * ex: Equal to the height of the letter 'x' in the font of the element on which it is used. This is useful when you want to size a container based on the height of a lower-case letter -- i.e. tighter -- rather than on the full size of the font (as you get with em). * ch: "Equal to the advance measure of the "0" (ZERO, U+0030) glyph found in the font used to render it." Since all digits in a font should be the same width, this unit is probably useful for pages that need to measure and render numbers in a reliable vertical alignment. * rem: The same as em but always returns the value for the root element of the page rather than the current element. Elements that use this unit will all scale against a common size, independently of the font-size of their contents. "There’s more to the CSS rem unit than font sizing" by Roman Rudenko has a lot more information and examples, as well as an explanation of how rem can stand in for the still nascent support for vw. ]]> https://www.earthli.com/news/view_article.php?id=2857 2013-06-29T17:00:15+02:00 https://earthli.com/users/marco

and is cross-published here. ------------------------------------------------------------------------ The article "Announcing the .NET Framework 4.5.1 Preview" provides an incredible amount of detail about a relatively exciting list of improvements for .NET developers. [x64 Edit & Continue] First and... ]]>

and is cross-published here. -------------------------------------------------------------------------------- The article "Announcing the .NET Framework 4.5.1 Preview" provides an incredible amount of detail about a relatively exciting list of improvements for .NET developers. [x64 Edit & Continue] First and foremost, the Edit-and-Continue feature is now available for x64 builds as well as x86 builds. Whereas an appropriate cynical reaction is that "it's about damn time they got that done", another appropriate reaction is to just be happy that they will finally support x64-debugging as a first-class feature in Visual Studio 2013. Now that they have feature-parity for all build types, they can move on to other issues in the debugger (see the list of suggestions at the end). [Async-aware debugging] We haven't had much opportunity to experience the drawbacks of the current debugger vis à vis asynchronous debugging, but the experience outlined in the call-stack screenshot below is one that is familiar to anyone who's done multi-threaded (or multi-fiber, etc.) programming. [image] Instead of showing the actual stack location in the thread within which the asynchronous operation is being executed, the new and improved version of the debugger shows a higher-level interpretation that places the current execution point within the context of the asnyc operation. This is much more in keeping with the philosophy of the async/await feature in .NET 4.5, which lets developers write asynchronous code in what appears to be a serial fashion. This improved readability has been translated to the debugger now, as well. [image] [Return-value inspection] The VS2013 debugger can now show the "direct return values and the values of embedded methods (the arguments)" for the current line. [1] Instead of manually selecting the text segment and using the Quick Watch window, you can now just see the chain of values in the "Autos" debugger pane. [image] [Nuget Improvements] "We are also releasing an update in Visual Studio 2013 Preview to provide better support for apps that indirectly depend on multiple versions of a single NuGet package. You can think of this as sane NuGet library versioning for desktop apps." We've been bitten by the afore-mentioned issue and are hopeful that the solution in Visual Studio 2013 will fill the gaps in the current release. The article describes several other improvements to the Nuget services, including integration with Windows Update for large-scale deployment. They also mentioned "a curated list of Microsoft .NET Framework NuGet Packages to help you discover these releases, published in OData format on the NuGet site", but don't mention whether the Nuget UI in VS2013 has been improved. The current UI, while not as awful and slow as initial versions, is still not very good for discovery and is quite clumsy for installation and maintenance. [User Voice for Visual Studio/.NET] You're not limited to just waiting on the sidelines to see which feature Microsoft has decided to implement in the latest version of .NET/Visual Studio. You should head over to the "User Voice for Visual Studio" site to get an account and vote for the issues you'd like the to work on next. Here's a list of the ones I found interesting, and some of which I've voted on. * Support Edit & Continue for all method bodies: While it is, to some degree, understandable that methods, fields and constants cannot be added or removed easily without restarting the debugger, method bodies can be modified. Unless they include lambdas or generics. Experience has shown that this means that we can't use Edit & Continue for most of our code. VS needs to "allow the modification of lambdas and anonymous methods" or the even higher-rated issue, "Debug Lambda expressions" . The other missing piece of this puzzle is to add "Support for generics under Edit & Continue" . * Better Generics and Design-by-Contract support: Language feature requests include "Expand Generic Constraints for constructors" , allow "[p]roper (generic) type ali[a]sing" and, near and dear to my heart, "Integrate Code Contracts more deeply in the .NET Framework" or "Integrate Code Contract Keywords into the main .Net Languages" . * There seem to be a lot of people asking Microsoft to work on "Silverlight 6" . Having developed several applications in Silverlight 5, we wouldn't be averse to seeing continued support rather than obsolescence for those projects. [2] * Make Visual Studio run faster and leaner: There are also pleas to "Decrease the Memory Footprint" (of Visual Studio) or "Make Visual Studio startup and shutdown faster" and "Make [Visual Studio] Solutions Load Faster" or at least "Improve performance of Visual Studio builds and other uses better when working with solutions with several projects (ex. 40+)" . Other ideas in this vein are add a "Visual Studio Task Manager" so we can at least see which components, plugins and panes are causing trouble. If all else fails, just "[m]ake VS scalable by switching to 64 bit" . -------------------------------------------------------------------------------- [1] In a similar vein, I found the issue "Bring back Classic Visual Basic, an improved version of VB6" to be interesting, simply because of the large number of votes for it (1712 at the time of writing). While it's understandable that VB6 developers don't understand the programming paradigm that came with the transition to .NET, the utterly reactionary desire to go back to VB6 is somewhat unfathomable. It's 2013, you can't put the dynamic/lambda/jitted genie back in the bottle. If you can't run with the big dogs, you'll have to stay on the porch...and stop being a developer. There isn't really any room for software written in a glorified batch language anymore. [1] This feature has been available for the unmanaged-code debugger (read: C++) for a while now. ]]> https://www.earthli.com/news/view_article.php?id=2849 2013-06-08T09:43:11+02:00 https://earthli.com/users/marco

and is cross-published here. ------------------------------------------------------------------------ Many improvements have been made to Microsoft's Entity Framework (EF) since I last used it in production code. In fact, we'd last used it waaaaaay back in 2008 and 2009 when EF had just been released.... ]]>

and is cross-published here. -------------------------------------------------------------------------------- Many improvements have been made to Microsoft's Entity Framework (EF) since I last used it in production code. In fact, we'd last used it waaaaaay back in 2008 and 2009 when EF had just been released. Instead of EF, I've been using the "Quino" ORM whenever I can. However, I've recently started working on a project where EF5 is used (EF6 is in the late stages of release, but is no longer generally available for production use). Though I'd been following the latest EF developments via the "ADO.Net blog" , I finally had a good excuse to become more familiar with the latest version with some hands-on experience. [Our history with EF] "Entity Framework: Be Prepared" was the first article I wrote about working with EF. It's quite long and documents the pain of using a 1.0 product from Microsoft. That version support only a database-first approach, the designer was slow and the ORM SQL-mapper was quite primitive. Most of the tips and advice in the linked article, while perhaps amusing, are no longer necessary (especially if you're using the Code-first approach, which is highly recommended). Our next update, "The Dark Side of Entity Framework: Mapping Enumerated Associations" , discusses a very specific issue related to mapping enumerated types in an entity model (something that Quino does very well). This shortcoming in EF has also "been addressed" but I haven't had a chance to test it yet. Our final article was on performance, "Pre-generating Entity Framework (EF) Views" , which, while still pertinent, no longer needs to be done manually (there's an "Entity Framework Power Tools" extension for that now). So let's just assume that that was the old EF; what's the latest and greatest version like? Well, as you may have suspected, you're not going to get an article about Code-first or database migrations. [1] While a lot of things have been fixed and streamlined to be not only much more intuitive but also work much more smoothly, there are still a few operations that aren't so intuitive (or that aren't supported by EF yet). [Standard way to delete objects] One such operation is deleting multiple objects in the database. It's not that it's not possible, but that the only solution that immediately appears is to, * load the objects to delete into memory, * then remove these objects from the context * and finally save changes to the context, which will remove them from the database The following code illustrates this pattern for a hypothetical list of users. var users = context.Users.Where(u => u.Name == "John"); foreach (var u in users) { context.Users.Remove(u); } context.SaveChanges(); This seems somewhat roundabout and quite inefficient. [2] [Support for batch deletes?] While the method above is fine for deleting a small number of objects -- and is quite useful when removing different types of objects from various collections -- it's not very useful for a large number of objects. Retrieving objects into memory only to delete them is neither intuitive nor logical. The question is: is there a way to tell EF to delete objects based on a query from the database? I found an example attached as an answer to the post "Simple delete query using EF Code First" . The gist of it is shown below. context.Database.SqlQuery( "DELETE FROM Users WHERE Name = @name", new [] { new SqlParameter("@name", "John") } ); To be clear right from the start, using ESQL is already sub-optimal because the identifiers are not statically checked. This query will cause a run-time error if the model changes so that the "Users" table no longer exists or the "Name" column no longer exists or is no longer a string. Since I hadn't found anything else more promising, though, I continued with this approach, aware that it might not be usable as a pattern because of the compile-time trade-off. Although the answer had four up-votes, it is not clear that either the author or any of his fans have actually tried to execute the code. The code above returns an IEnumerable but doesn't actually do anything. After I'd realized this, I went to MSDN for more information on the SqlQuery method. The documentation is not encouraging for our purposes (still trying to delete objects without first loading them), as it describes the method as follows (emphasis added), "Creates a raw SQL query that will return elements of the given generic type. The type can be any type that has properties that match the names of the columns returned from the query, or can be a simple primitive type." This does not bode well for deleting objects using this method. Creating an enumerable does very little, though. In order to actually execute the query, you have to evaluate it. Die Hoffnung stirbt zuletzt [3] as we like to say on this side of the pond, so I tried evaluating the enumerable. A foreach should do the trick. var users = context.Database.SqlQuery( "DELETE FROM Users WHERE Name = @name", new [] { new SqlParameter("@name", "John") } ); foreach (var u in users) { // NOP? } As indicated by the "NOP?" comment, it's unclear what one should actually do in this loop because the query already includes the command to delete the selected objects. Our hopes are finally extinguished with the following error message: System.Data.EntityCommandExecutionException : The data reader is incompatible with the specified 'Demo.User'. A member of the type, 'Id', does not have a corresponding column in the data reader with the same name. That this approach does not work is actually a relief because it would have been far too obtuse and confusing to use in production. It turns out that the SqlQuery only works with SELECT statements, as was strongly implied by the documentation. var users = context.Database.SqlQuery( "SELECT * FROM Users WHERE Name = @name", new [] { new SqlParameter("@name", "John") } ); Once we've converted to this syntax, though, we can just use the much clearer and compile-time--checked version that we started with, repeated below. var users = context.Users.Where(u => u.Name == "John"); foreach (var u in users) { context.Users.Remove(u); } context.SaveChanges(); So we're back where we started, but perhaps a little wiser for having tried. [Deleting objects with Quino] As a final footnote, I just want to point out how you would perform multiple deletes with the Quino ORM. It's quite simple, really. Any query that you can use to select objects you can also use to delete objects [4]. So, how would I execute the query above in Quino? Session.Delete(Session.CreateQuery().WhereEquals(User.MetaProperties.Name, "John").Query); To make it a little clearer instead of showing off with a one-liner: var query = Session.CreateQuery(); query.WhereEquals(User.MetaProperties.Name, "John"); Session.Delete(query); Quino doesn't support using Linq to create queries, but its query API is still more statically checked than ESQL. You can see how the query could easily be extended to restrict on much more complex conditions, even including fields on joined tables. -------------------------------------------------------------------------------- [1] As I wrote, We're using Code-first, which is much more comfortable than using the database-diagram editor of old. We're also using the nascent "Migrations" support, which has so far worked OK, though it's nowhere near as convenient as Quino's automated schema-migration. [1] Though it is inefficient, it's better than a lot of other examples out there, which almost unilaterally include the call to context.SaveChanges() inside the foreach-loop. Doing so is wasteful and does not give EF an opportunity to optimize the delete calls into a single SQL statement (see footnote below). [1] Translates to: "Hope is the last (thing) to die." [1] With the following caveats, which generally apply to all queries with any ORM: * Many databases use a different syntax and provide different support for DELETE vs. SELECT operations. * Therefore, it is more likely that more complex conditions are not supported for DELETE operations on some database back-ends * Since the syntax often differs, it's more likely that a more complex query will fail to map properly in a DELETE operation than in a SELECT operation simply because that particular combination has never come up before. * That said, Quino has quite good support for deleting objects with restrictions not only on the table from which to delete data but also from other, joined tables. Some combination of these reasons possibly accounts for EF's lack of support for batch deletes. ]]> https://www.earthli.com/news/view_article.php?id=2823 2013-05-05T21:36:02+02:00 https://earthli.com/users/marco

Works on file 1 Team Two - branch two > Works on file 1 Team One promotes file 1 into the Master B branch, there are some conflicts that they are working out but the file is promoted." I originally answered that I wasn't sure what it meant to "promote" a file while still working on it. How can a file be commited or checked in without having resolved all of the conflicts? As it turns out, it can't. As documented in "TFS Server 2012 and Promoting changes" , promotion simply means telling TFS to pick up local changes and add them to the list of "Pending Changes". This is part of a new TFS2012 feature called "Local Workspaces". A promoted change corresponds to having added a file to a change list in Perforce or having staged a file in Git. The net effect, though, is that the change is purely local. That is has been promoted has nothing to do with merging or committing to the shared repository. Other users cannot see your promoted changes. When you pull down new changes from the server, conflicts with local "promoted" changes will be indicated as usual, even if TFS has already indicated conflicts between a previous change and another promoted, uncommitted version of the same file. Any other behavior else would be madness. [1] "Team Two checks in their file 1 into the Master B branch. They back out the changes that Team One made without telling anyone anything." There's your problem. This should never happen unless Team Two has truly determined that their changes have replaced all of the work that Team One did or otherwise made it obsolete. If people don't know how to deal with merges, then they should not be merging. Just as Stevie Wonder's not allowed behind the wheel of a car, neither should some developers be allowed to deal with merge conflicts. In my opinion, though, any developer who can't deal with merges in code that he or she is working on should be moved another team or, possibly, job. You have to know your own code and you have to know your tools. [2] "Team One figures out the conflicts in their branch and re-promotes file one (and other files) to Master B branch. The source control system remembers that file 1 was backed out by Team Two so it doesn't promote file 1 but doesn't let the user know." This sounds insane. When a file is promoted -- i.e. added to the pending changes -- it is assumed that the current version is added to the pending changes, akin to staging a file in Git. When further changes are made to the file locally, the source-control system should indicate that it has changed since having been promoted (i.e. staged). When you re-promote the file (re-stage it), TFS should treat that as the most recent version in your workspace. When you pull down the changes from Team 2, you will have all-new conflicts to resolve because your newly promoted file will still be in conflict with the changes they made to "file 1" -- namely that they threw away all of the changes that you'd made previously. And, I'm not sure how it works in TFS, but in Git, you can't "back out" a commit without leaving a trail: * Either there is a merge commit where you can see that Team Two chose to "accept their version" rather than "merge" or "accept other version" * Or, there is a "revert" commit that "undoes" the changes from a previous commit Either way, your local changes will cause a conflict because they will have altered the same file in the same place as either the "merge" or "revert" commit and -- this is important -- will have done so after that other commit. To recap, let me summarize what this sounds like: * T1: I want to check in file1 * TFS: You have conflicts * T1: Promote file1 so that TFS knows about (other users can't see it yet because it hasn't been committed) * TFS: Okie dokie * T2: I want to check in file1 * TFS: You have conflicts * T2: Fuck that. Use my version. Oh, and, fuck T1. * TFS: I hear and obey. T2/file1 it is. * T1: OK, I resolved conflicts; here's the final version of file1 * TFS: Thanks! *tosses T1/file1 out the window* I don't believe that this is really possible -- even with TFS -- but, if this is a possibility with your source-control, then you have two problems: 1. You have team members who don't know how to merge 2. Your source control is helping them torpedo development There is probably a setting in your source-control system that disallows simultaneous editing for files. This is a pretty huge restriction, but if your developers either can't or won't play nice, you probably have no choice. -------------------------------------------------------------------------------- [1] This is not to rule out such behavior 100%, especially in a source-control system with which I am largely unfamiliar. It only serves to indicate the degree to which I would be unwilling to work with any system that exhibits this kind of behavior. [1] Different companies can have different grace periods for learning these two things, of course. I suppose that grace period can be interminably long, but... ]]> https://www.earthli.com/news/view_article.php?id=2784 2013-02-12T21:44:37+01:00 https://earthli.com/users/marco

is quite long (26 pages), detailed and involved. To be frank, most of the notation was foreign to me -- to say nothing of making heads or tails of most of the proofs and lemmas -- but I found the higher-level discussions and... ]]>

is quite long (26 pages), detailed and involved. To be frank, most of the notation was foreign to me -- to say nothing of making heads or tails of most of the proofs and lemmas -- but I found the higher-level discussions and conclusions quite interesting. The abstract is concise and describes the project very well: "A key challenge for concurrent programming is that side-effects (memory operations) in one thread can affect the behavior of another thread. In this paper, we present a type system to restrict the updates to memory to prevent these unintended side-effects. We provide a novel combination of immutable and unique (isolated) types that ensures safe parallelism (race freedom and deterministic execution). The type system includes support for polymorphism over type qualifiers, and can easily create cycles of immutable objects. Key to the system's flexibility is the ability to recover immutable or externally unique references after violating uniqueness without any explicit alias tracking. Our type system models a prototype extension to C# that is in active use by a Microsoft team. We describe their experiences building large systems with this extension. We prove the soundness of the type system by an embedding into a program logic." The project proposes a type-system extension with which developers can write provably safe parallel programs -- i.e. "race freedom and deterministic execution" -- with the amount of actual parallelism determined when the program is analyzed and compiled rather than decided by a programmer creating threads of execution. [Isolating objects for parallelism] The "isolation" part of this type system reminds me a bit of the way that SCOOP addresses concurrency. That system also allows programs to designate objects as "separate" from other objects while also releasing the program from the onus of actually creating and managing separate execution contexts. That is, the syntax of the language allows a program to be written in a provably correct way (at least as far as parallelism is concerned; see the "other provable-language projects" section below). In order to execute such a program, the runtime loads not just the program but also another file that specifies the available virtual processors (commonly mapped to threads). Sections of code marked as "separate" can be run in parallel, depending on the available number of virtual processors. Otherwise, the program runs serially. In SCOOP, methods are used as a natural isolation barrier, with input parameters marked as "separate". See "SCOOP: Concurrency for Eiffel" and "SCOOP (software)" for more details. The paper also contains an entire section listing other projects -- many implemented on the the JVM -- that have attempted to make provably safe programming languages. The system described in this paper goes much further, adding immutability as well as isolation (the same concept as "separate" in SCOOP). An interesting extension to the type system is that isolated object trees are free to have references to immutable objects (since those can't negatively impact parallelism). This allows for globally shared immutable state and reduces argument-passing significantly. Additionally, there are readable and writable references: the former can only be read but may be modified by other objects (otherwise it would be immutable); the latter can be read and written and is equivalent to a "normal" object in C# today. In fact, "[...] writable is the default annotation, so any single-threaded C# that does not access global state also compiles with the prototype." [Permission types] In this safe-parallel extension, a standard type system is extended so that every type can be assigned such a permission and there is "support for polymorphism over type qualifiers", which means that the extended type system includes the permission in the type, so that, given B => A, a reference to readable B can be passed to a method that expects an immutable A. In addition, covariance is also supported for generic parameter types. When they say that the "[k]ey to the system's flexibility is the ability to recover immutable or externally unique references after violating uniqueness without any explicit alias tracking", they mean that the type system allows programs to specify sections that accept isolated references as input, lets them convert to writable references and then convert back to isolated objects -- all without losing provably safe parallelism. This is quite a feat since it allows programs to benefit from isolation, immutability and provably safe parallelism without significantly changing common programming practice. In essence, it suffices to decorate variables and method parameters with these permission extensions to modify the types and let the compiler guide you as to further changes that need to be made. That is, an input parameter for a method will be marked as immutable so that it won't be changed and subsequent misuse has to be corrected. Even better, they found that, in practice, it is possible to use extension methods to allow parallel and standard implementations of collections (lists, maps, etc.) to share most code. "A fully polymorphic version of a map() method for a collection can coexist with a parallelized version pmap() specialized for immutable or readable collections. [...] Note that the parallelized version can still be used with writable collections through subtyping and framing as long as the mapped operation is pure; no duplication or creation of an additional collection just for concurrency is needed." [Real projects and performance impact] Much of the paper is naturally concerned with proving that their type system actually does what it says it does. As mentioned above, at least 2/3 of the paper is devoted to lemmas and large swaths of notation. For programmers, the more interesting part is the penultimate section that discusses the extension to C# and the experiences in using it for larger projects. "A source-level variant of this system, as an extension to C#, is in use by a large project at Microsoft, as their primary programming language. The group has written several million lines of code, including: core libraries (including collections with polymorphism over element permissions and data-parallel operations when safe), a webserver, a high level optimizing compiler, and an MPEG decoder." Several million lines of code is, well, it's an enormous amount of code. I'm not sure how many programmers they have or how they're counting lines or how efficiently they write their code, but millions of lines of code suggests generated code of some kind. Still, taken with the next statement on performance, that much code more than proves that the type system is viable. "These and other applications written in the source language are performance-competitive with established implementations on standard benchmarks; we mention this not because our language design is focused on performance, but merely to point out that heavy use of reference immutability, including removing mutable static/global state, has not come at the cost of performance in the experience of the Microsoft team." Not only is performance not impacted, but the nature of the typing extensions allows the compiler to know much more about which values and collections can be changed, which affects how aggressively this data can be cached or inlined. "In fact, the prototype compiler exploits reference immutability information for a number of otherwise-unavailable compiler optimizations. [...] Reference immutability enables some new optimizations in the compiler and runtime system. For example, the concurrent GC can use weaker read barriers for immutable data. The compiler can perform more code motion and caching, and an MSIL-to-native pass can freeze immutable data into the binary." [Incremental integration ("unstrict" blocks)] In the current implementation, there is an unstrict block that allows the team at Microsoft to temporarily turn off the new type system and to ignore safety checks. This is a pragmatic approach which allows the software to be run before it has been proven 100% parallel-safe. This is still better than having no provably safe blocks at all. Their goal is naturally to remove as many of these blocks as possible -- and, in fact, this requirement drives further refinement of the type system and library. "We continue to work on driving the number of unstrict blocks as low as possible without over-complicating the type system’s use or implementation." The project is still a work-in-progress but has seen quite a few iterations, which is promising. The paper was written in 2012; it would be very interesting to take it for a test drive in a CTP. [Other provable-language projects] A related project at Microsoft Research "Spec#" contributed a lot of basic knowledge about provable programs. The authors even state that the "[...] type system grew naturally from a series of efforts at safe parallelism. [...] The earliest version was simply copying Spec#’s [Pure] method attribute, along with a set of carefully designed task-and data-parallelism libraries." Spec#, in turn, is a "[...] formal language for API contracts (influenced by JML, AsmL, and Eiffel), which extends C# with constructs for non-null types, preconditions, postconditions, and object invariants". Though the implementation of this permissions-based type system may have started with Spec#, the primary focus of that project was more a valiant attempt to bring Design-by-Contract principles ("examples and some discussion here" ) to the .NET world via C#. Though spec# has "downloadable code" , the project hasn't really been updated in years. This is a shame, as support for "Eiffel" [1] in .NET, mentioned above as one of the key influences of spec#, was dropped by ISE Eiffel long ago. Spec#, in turn, was mostly replaced by Microsoft Research's "Contracts" project (an older version of which was covered in depth in "Microsoft Code Contracts: Not with a Ten-foot Pole" ). The Contracts project seems to be alive and well: the most recent release is from October, 2012. I have not checked it out since my initial thumbs-down review (linked above) but did note in passing that the implementation is still (A) library-only and (B) does not support Visual Studio 2012. The library-only restriction is particularly galling, as such an implementation can lead to repeated code and unwieldy anti-patterns. As documented in the "Contracts FAQ" , the current implementation of the "tools take care of enforcing consistency and proper inheritance of contracts" but this is presumably accomplished with compiler errors that require the programmer to include contracts from base methods in overrides. The seminal work Object-oriented Software Construction by Bertrand Meyer (vol. II in particular) goes into tremendous detail on a type system that incorporates contracts directly. The type system discussed in this article covers only parallel safety: null-safety and other contracts are not covered at all. If you're at all interested in these types of language extensions, the vol.2 of OOSC is a great read. The examples are all in "Eiffel" but should be relatively accessible. Though some features -- generics, notably but also tuples, once routines and "agents" -- have since made their way into C# and other more commonly used languages, many others -- such as contracts, anchored types (contravariance is far too constrained in C# to allow them), covariant return types, covariance everywhere, multiple inheritance, explicit feature removal, loop variants and invariants, etc. -- are still not available. Subsequent interesting work has also been done on extensions that allow creation of "provably null-safe programs" , something also addressed in part by Microsoft Research's "Contracts" project. ]]> https://www.earthli.com/news/view_article.php?id=2780 2013-02-03T23:04:09+01:00 https://earthli.com/users/marco

is one such -- explaining better than ever why rewriting history is better than server-side, immutable commits. In the article cited above, Sandofsky divides his work up into "Short-lived work [...] larger work [...] and branch bankrupty." These concepts are documented to some degree in the Branch Management chapter of the "Encodo Git Handbook" (of which I am co-author). I will expand on these themes below. Note: The linked articles deal exclusively with the command line, which isn't everyone's favorite user interface (I, for one, like it). We use the "SmartGit/Hg" client for visualizing diffs, organizing commits and browsing the log. We also use the command-line for a lot of operations, but SmartGit is a very nice tool and version 3 supports nearly all of the operations described in this article. [What is rebasing?] As you can see from the diagram above, a well-organized and active project will have multiple branches. Merging and rebasing are two different ways of getting commits from one branch into another. Merging commits into a branch creates a merge commit, which shows up in the history to indicate that n commits were made on a separate branch. Rebasing those commits instead re-applies those commits to the head of the indicated branch without a merge commit. In both cases there can be conflicts, but one method doesn't pose a greatest risk of them than the other. [4] You cannot tell from the history that rebased commits were developed in a separate branch. You can, however, tell that the commits were rebased because the author date (the time the commit was originally created) differs from the commit date (the last time that the commit was applied). [What do you recommend?] At "Encodo" , we primarily work in the master branch because we generally work on very manageable, bite-sized issues that can easily be managed in a day. Developers are free to use local branches but are not required to do so. If some other requirement demands priority, we shunt the pending issue into a private branch. Such single-issue branches are focused and involve only a handful of files. It is not at all important to "remember" that the issue was developed in a branch rather than the master branch. If there are several commits, it may be important for other users to know that they were developed together and a merge-commit can be used to indicate this. Naturally, larger changes are developed in feature branches, but those are generally the exception rather than the rule. "Remember: Nowhere in those requirements does it say that you're not allowed to lie about how you got to that pristine tree of commits." Otherwise? Local commit history is absolutely not sacrosanct. We rebase like crazy to avoid unwanted merge commits. That is, when we pull from the central repository, we rebase our local commits on top of the commits that come form the origin. This has worked well for us. If the local commit history is confusing -- and this will sometimes come up during the code review -- we use an interactive rebase to reorganize the files into a more soothing and/or understandable set of commits. See "Sandofsky's article" for a good introduction to using interactive rebasing to combine and edit commits. Naturally, we weigh the amount of confusion caused by the offending commits against the amount of effort required to clean up the history. We don't use bisect [5] very often, so we don't invest a lot of time in enforcing the clean, compilable commits required by that tool. For us, the history is interesting, but we rarely go back farther than a few weeks in the log. [6] [When to merge? When to rebase?] At Encodo, there are only a few reasons to retain a merge commit in the official history: 1. If we want to remember which commits belonged to a particular feature. Any reasonable tool will show these commits graphically as a separate strand running alongside the master branch. 2. If a rebase involves too much effort or is too error-prone. If there are a lot of commits in the branch to be integrated, there may be subtle conflicts that resolve more easily if you merge rather than rebase. Sometimes we just pull the e-brake and do a merge rather than waste time and effort trying to get a clean rebase. This is not to say that the tools are lacking or at fault but that we are pragmatic rather than ideological. [7] 3. If there are merge commits in a feature branch with a large number of well-organized commits and/or a large number of changes or affected files. In this case, using a squash merge and rebuilding the commit history would be onerous and error-prone, so we just merge to avoid issues that can arise when rebasing merge commits (related to the point above). [When should I use private branches? What are they exactly?] There are no rules for local branches: you can name them whatever you like. However, if you promote a local branch to a private branch, at Encodo we use the developer's initials as the prefix for the branch. My branches are marked as "mvb/feature1", for example. What's the difference between the two? Private branches may get pushed to our common repository. Why would you need to do that? Well, I, for example, have a desktop at work and, if I want to work at home, I have to transfer my workspace somehow to the machine at home. One solution is to work on a virtual machine that's accessible to both places; another is to remote in to the desktop at work from home; the final one is to just push that work to the central repository and pull it from home. The offline solution has the advantage of speed and less reliance on connectivity. What often happens to me is that I start work on a feature but can only spend an hour or two on it before I get pulled off onto something else. I push the private branch, work on it a bit more at home, push back, work on another, higher-priority feature branch, merge that in to master, work on master, whatever. A few weeks later and I've got a private branch with a few ugly commits, some useful changes and a handful of merge commits from the master branch. The commit history is a disgusting mess and I have a sneaking suspicion that I've only made changes to about a dozen files but have a dozen commits for those changes. That's where the aforementioned "branch bankruptcy" comes in. You're not obligated to keep that branch; you can keep the changes, though. As shown in the "referenced article" , you execute the following git commands: git checkout master git checkout -b cleaned_up_branch git merge --squash private_feature_branch git reset The --squash tells git to squash all of the changes from the private_feature_branch into the index (staging) and reset the index so that those changes are in the working tree. From here, you can make a single, clean, well-written commit or several commits that correspond logically to the various changes you made. Git also lets you lose your attachment to checking in all the changes in a file at once: if a file has changes that correspond to different commits, you can add only selected differences in a file to the index (staging). "In praise of Git’s index" by Aristotle Pagaltzis provides a great introduction. If you, like me, regularly take advantage of refactoring and cleanup tools while working on something else, you'll appreciate the ability to avoid checking in dozens of no-brainer cleanup/refactoring changes along with a one-liner bug-fix. [8] [One final example: cherry picking and squashing] I recently renamed several projects in our solution, which involved renaming the folders as well as the project files and all references to those files and folders. Git automatically recognizes these kind of renames as long as the old file is removed and the new file is added in the same commit. I selected all of the files for the rename in SmartGit and committed them, using the index editor to stage only the hunks from the project files that corresponded to the rename. Nice and neat. I selected a few other files and committed those as a separate bug-fix. Two seconds later, the UI refreshed and showed me a large number of deleted files that I should have included in the first commit. Now, one way to go about fixing this is to revert the two commits and start all over, picking the changes apart (including playing with the index editor to stage individual hunks). Instead of doing that, I did the following: 1. I committed the deleted files with the commit message "doh!" (to avoid losing these changes in the reset in step 3) 2. I created a "temp" branch to mark that commit (to keep the commit visible once I reset in step 3) 3. I hard-reset my master branch to the origin 4. I cherry-picked the partial-rename commit to the workspace 5. I cherry-picked the "doh!" commit to the workspace 6. Now the workspace had the rename commit I'd wanted in the first place 7. I committed that with the original commit message 8. I cherry-picked and committed the separate bug-fix commit 9. I deleted the "temp" branch (releasing the incorrect commits on it to be garbage-collected at some point) Now my master branch was ready to push to the server, all neat and tidy. And nobody was the wiser. -------------------------------------------------------------------------------- [1] There are alternatives now, though, like Microsoft's TypeScript, that warrant a look if only because they help tighten the error-finding feedback loop and have the potential to make you more efficient (the efficiency may be robbed immediately back, however, if debugging generated code becomes difficult or even nightmarish). [1] Once you've pushed, though? No touchie. At that point, you've handed in your test and you get graded on that. [1] According to my business card, I'm a "senior developer and partner" at "Encodo System AG" . [1] With the exception, mentioned elsewhere as well, that rebasing merge-commits can sometimes require you to re-resolve previously resolved conflicts, which can be error-prone if the conflicts were difficult to resolve in the first place. Merging merge-commits avoids this problem. [1] bisect is a git feature that executes a command against various commits to try to localize the commit that caused a build or test failure. Basically, you tell it the last commit that worked and git uses a binary search to find the offending commit. Of course, if you have commits that don't compile, this won't work very well. We haven't used this feature very much because we know the code in our repositories well and using blame and log is much faster. Bisect is much more useful for maintainers that don't know the code very well, but still need to figure out at which commit it stopped working. [1] To be clear: we're only so cavalier with our private repositories to which access is restricted to those who already know what's going on. If we commit changes to public, open-source or customer repositories, we make sure that every commit compiles. See "Aristotle's index article" (cited above) for tips on building and testing against staged files to ensure that a project compiles, runs and passes all tests before making a commit -- even if you're not committing all extant changes. [1] That said, with experience we've learned that an interactive rebase and judicious squashing will create commits that avoid these problems. With practice, these situations crop up more and more rarely. [1] Of course, you can also create a separate branch for your refactoring and merge it all back together, but that's more work and is in my experience rarely necessary. ]]> https://www.earthli.com/news/view_article.php?id=758 2012-11-22T23:24:02+01:00 https://earthli.com/users/marco

db->logged_query ("SELECT ... ]]>

db->logged_query ("SELECT folder_id FROM" . $this->app->table_names->objects . "WHERE id = $obj->object_id"); if ($this->db->next_record ()) $folder_id = $this->db->f ("folder_id"); else // raise exception? ignore? what to do? } Above we see a situation in which you may decide against a stricter enforcement because whereas the error is clear, the reaction is not. This is also a big part of working with contracts: deferring reactions. Often -- especially when developing libraries -- you're in code so deep that the desired reaction could be one of many depending on the deployment of that code. The code above is taken from the publishing loop in the "webcore" ; it's used to publish comments. In effect, the code has detected that a comment object id has been passed in that doesn't correspond to anything in the system. It's bogus. It's wrong. Some deployments -- I would hazard most -- would just like to silently ignore the error and publish as much as possible. Silently ignoring an error will always bite you in the ass in the end (pun intended). The key here is that whereas the person deploying the final system should be perfectly free to ignore the error, you, as the library developer, can and must not. Let's see what kind of reactions we could have here. Well, isn't that what exceptions were invented for? They're for transmitting error conditions up out of deep library code. Problem solved. For more severe errors in which the code cannot continue, the answer is quite clear: you simply throw an exception. However, in the situation above, it's not so clear. The problem is easily skipped and most of the rest of the job can be finished. Here is where deferral comes in. Just call a function that will handle it later. This function can log the error or warning, display it to the user, ask to abort/retry/ignore, consult a table for same, throw an exception or just ignore it. It's not your problem to dispatch solutions to encountered errors. It's your job to detect them and maintain the integrity of the running code. Simply throwing an exception no matter what the error condition is, in effect, making a decision about how the error will be handled. Control is lost because the exception handler is necessarily higher up. This is a bad thing if you'd actually like the code to do the best it can. As any experience at all will have shown you, some errors are just warnings or hints. It's not just black and white, error or not. Many deployments of the system containing the code above will actually treat the issue as a warning and log it for the database techs to address. However, to assume the opposite, that callers want errors to be swallowed, cheats those callers as well. It cheats them because it becomes incredibly hard to find errors; they must be detected by subtle logic or data problems (e.g. Hmmm...the log shows it only sent 500 emails, I thought there were 503 subscribers...). If the system never complains or logs anything, the end user calls you first. It cheats you because you can never adequately test your system because it never complains. Everything's OK. It kind of works. It mostly works. The desire for safety or avoiding crashes or exceptions on the client side should never override the desire to have correct code that detects error conditions. If you write library code or end-user code, that code deals mostly with detecting and reporting misused functions. The functionality itself is generally straightforward; it's wrapping the interface around it that's hard. The only thing that you'll probably spend more time on is hunting down memory bugs -- and, yes, even if you're using a garbage-collected runtime, you can still have memory bugs. What would you call it if the memory required by your publication script increases in proportion to the number of subscribers and mails? If you're wondering what I ended up doing in the case above, I decided on a function call 'raise'. It sounds like an exception, and that's usually what happens: it breaks the code on that line, but a little more elegantly than the usual PHP die statement. While the default handler for exceptions simply issues a fancy die statement, that handler can be replaced with a different one, one that redirects to an HTML page with a nicely formatted error printout and a form for submitting the error. Since this code is likely to run inside a script that just wants to send subscriptions and doesn't care about data integrity errors, the handler would probably be replaced with something that suppresses the exception, but logs the error. That way, once the subscription run is done, you can view the error log and see if there are data integrity problems, and, perhaps more importantly, you see them all at once instead of just one at a time. And, more importantly still, those subscribers for whom there were no problems received their mail on time. ]]> https://www.earthli.com/news/view_article.php?id=2720 2012-11-22T19:45:47+01:00 https://earthli.com/users/marco

and is cross-published here. ------------------------------------------------------------------------ In the latest version of Quino -- version 1.8.5 -- we took a long, hard look at the patterns we were using to create metadata. The metadata for an application includes all of the usual Quino stuff:... ]]>

and is cross-published here. -------------------------------------------------------------------------------- In the latest version of Quino -- version 1.8.5 -- we took a long, hard look at the patterns we were using to create metadata. The metadata for an application includes all of the usual Quino stuff: classes, properties, paths, relations. With each version, though we're able to use the metadata in more places. That means that the metadata definition code grows and grows. We needed some way to keep a decent overview of that metadata without causing too much pain when defining it. In order to provide some background, the following are the high-level requirements that we kept in mind while designing the new pattern and supporting framework. Manage complexity A simple model should be easy and straightforward to build, with no cumbersome boilerplate; complex models should support multiple layers and provide an overview Leverage existing knowhow Our users don't want to learn a new language/IDE in order to create metadata; neither do we want to provide support for our own metadata-definition language Support modularization Modules can be used to hide complexity but are also sometimes necessary to define hard boundaries in the application metadata Support extensibility Interdependent modules and overlays will need to refer to elements in other modules; there needs to be a standard mechanism for defining and accessing metadata elements that doesn't rely on string constants [1] Support refactoring Rely on convention and name-matching as little as possible to avoid subtle errors Support introspection Developers that stick to the pattern should be able to maximize efficiency using common navigation and introspection [2] tools like Visual Studio or ReSharper. [Definition Language] Quino metadata has always been defined using a .NET language -- in our case, we always use C# to define the metadata, using the MetaBuilder or InMemoryMetaBuilder to compose the application model. This approach satisfies the need to leverage existing tools, refactoring and introspection. Since Quino metadata is an in-memory construct, there will always be a .NET API for creating metadata. This is not to say that there will never be a DSL to define Quino metadata but that such an approach is not the subject of this post. [Modularization] Quino applications have always been able to define and integrate metadata modules (e.g. reporting or security) using an IMetaModuleBuilder. Modules solved interdependency issues by splitting the metadata-generation into several phases: * Add classes and foreign keys * Add paths between classes (depends on foreign keys) * Add calculated properties and relations (depends on paths) * Add layouts (depends on all properties) In this way, when a module needed to add a path between a class that it had defined and a class defined in another module, it could be guaranteed that classes and foreign keys for all modules had been defined before any paths were created. Likewise for classes that wanted to define relations based on paths defined in other modules. The limitation of the previous implementation was that a module generator always created its own module and builder and could not simply re-use those created by another generator. Basically, there was no "lightweight" way of splitting metadata-generation into separate files for purely organizational purposes. There were also a few issues with the implementation of the main model-generation code as well. The previous pattern depended heavily on local variables, all defined within one mammoth function. Separating code into individual method calls was ad-hoc -- each project did it a little differently -- and involved a lot of migration of local variables to instance variables. With all code in a single method, file-structure navigation tools couldn't help at all. The previous pattern prescribed using file comments or regions that could be located using "find in file". This was clearly sub-optimal. [The new pattern] The new pattern that can be applied for all models, bit or small includes the following parts: Model generator As before, there is a class that implements the IMetaModelGenerator interface. This class is used by the application configuration and various tools (e.g. the code generator or UML generator) to create the model. Model elements Metadata that is referenced from multiple steps in the metadata-generation process is stored in a separate object (or objects) called the model elements. (E.g. classes are created in the AddClasses() step and referenced in the AddPaths, AddProperties and AddLayouts steps.) The model elements typically has two properties called Classes and Paths. Metadata generators Module generators still exist, but there are now also metadata generators that are lightweight, using a metadata builder and elements defined by another generator (typically a module generator or the model generator itself). This may sound like a lot of overhead for a simple application, but it's really not that much extra code. The benefits are: * Models, modules and lightweight parts all use the same pattern, with the same phases and method names. That makes it far easier to know where to look for a definition * Since the pattern is the same, it's easy to move functionality from one module to another or to split one module into multiple lightweight parts without doing a lot of refactoring * A small model will naturally grow to a medium or large model, all while using the same pattern. There is no moment during development where you have to do a major refactoring in order to get organized: the pattern will naturally support a clean coding style. [Building a model, step by step] But enough chatter; let's take a look at the absolute minimum boilerplate for an empty model. [Step zero: create the boilerplate] public class DemoModelElements { public DemoModelElements() { Classes = new DemoModelClasses(); Paths = new DemoModelPaths(); } public DemoModelClasses Classes { get; private set; } public DemoModelPaths Paths { get; private set; } } public class DemoModelPaths { } public class DemoModelClasses { } public class DemoCoreGenerator : DependentMetadataGeneratorBase { } public class DemoModelGenerator : MetaBuilderBasedModelGeneratorBase { protected override void AddMetadata() { Builder.Include(); } } The code above is functional but doesn't actually create any metadata. So what does it do? 1. It uses the generic MetaBuilderBasedModelGeneratorBase to indicate the type of Elements that will be exposed by this model generator. The elements class is created automatically and is available as the property Elements (as we'll see in the examples below). Additionally, we're using a ModelGeneratorBase that is based on a MetaBuilder which means that the property Builder is also available and is of type MetaBuilder. 2. It includes the DemoCoreGenerator which is a dependent generator -- it's lightweight and uses the elements and builder from its owner. The exact types are shown in the class declaration; it can be read as: get elements of type DemoModelElements and a builder of type MetaBuilder from the generator with type DemoModelGenerator. The initial generic argument can be any other metadata generator that implements the IElementsProvider interface. 3. The model generator overrides AddMetadata to include the metadata created by DemoCoreGenerator in the model. Even though it's not very much code, you can create a snippet or a file template with Visual Studio or a Live Template or file template with ReSharper to quickly create a new model. [Step one: define the model] Now, let's fill the empty model with some metadata. The first step is to define the model that we're going to build. That part goes in the AddMetadata() method. [3] public class DemoModelGenerator : MetaBuilderBasedModelGeneratorBase { protected override void AddMetadata() { Builder.CreateModel("Demo", /*Guid*/); Builder.CreateMainModule("Encodo.Quino"); Builder.Include(); } } [Step two: add a class] A typical next step is to define a class. Let's do that. public class DemoModelClasses { public IMetaClass Company { get; set; } } public class DemoCoreGenerator : DependentMetadataGeneratorBase { protected override void AddClasses() { Elements.Classes.Company = Builder.AddClassWithDefaultPrimaryKey("Company", /*Guid*/, /*Guid*/); } } As you can see, we added a new class to the elements and created and assigned it in the AddClasses() phase of metadata-generation. [Step three: add another class and a path] An obvious next step is to create another class and define a path between them. public class DemoModelClasses { public IMetaClass Company { get; set; } public IMetaClass Person { get; set; } } public class DemoCoreGenerator : DependentMetadataGeneratorBase { protected override void AddClasses() { Elements.Classes.Company = Builder.AddClassWithDefaultPrimaryKey("Company", /*Guid*/, /*Guid*/); Elements.Classes.Person = Builder.AddClassWithDefaultPrimaryKey("Person", /*Guid*/, /*Guid*/); Builder.AddInvisibleProperty(Elements.Classes.Person, "CompanyId", MetaType.Key, true, /*Guid*/); } protected override void AddPaths() { Elements.Paths.CompanyPersonPath = Builder.AddOneToManyPath( Elements.Classes.Company, "Id", Elements.Classes.Person, "CompanyId", /*Guid*/, /*Guid*/ ); } } [Step four: add relations] Having a path is not enough, though. We can also define how the relations on that path are exposed in the classes. public class DemoCoreGenerator : DependentMetadataGeneratorBase { protected override void AddClasses() { Elements.Classes.Company = Builder.AddClassWithDefaultPrimaryKey("Company", /*Guid*/, /*Guid*/); Elements.Classes.Person = Builder.AddClassWithDefaultPrimaryKey("Person", /*Guid*/, /*Guid*/); Builder.AddInvisibleProperty(Elements.Classes.Person, "CompanyId", MetaType.Key, true, /*Guid*/); } protected override void AddPaths() { Elements.Paths.CompanyPersonPath = Builder.AddOneToManyPath( Elements.Classes.Company, "Id", Elements.Classes.Person, "CompanyId", /*Guid*/, /*Guid*/ ); } protected override void AddProperties() { Builder.AddRelation(Elements.Classes.Company, "People", "", Elements.Paths.CompanyPersonPath); Builder.AddRelation(Elements.Classes.Person, "Company", "", Elements.Paths.CompanyPersonPath); } } OK, now we have a model with two entities -- companies and people -- that are related to each other so that a company has a list of people and each person belongs to a company. [Step five: add translations] Now we'd like to make the metadata support German as well as English. Quino naturally supports more generalized ways of doing this (e.g. importing from files), but let's just add the metadata manually to see what that would look like (unaffected methods are left off for brevity). public class DemoModelElements { public DemoModelElements() { Classes = new DemoModelClasses(); Paths = new DemoModelPaths(); } public ILanguage English { get; set; } public ILanguage German { get; set; } public DemoModelClasses Classes { get; private set; } public DemoModelPaths Paths { get; private set; } } public class DemoCoreGenerator : DependentMetadataGeneratorBase { protected override void AddCoreElements() { Elements.English = Builder.AddDisplayLanguage("en-US", "English"); Elements.German = Builder.AddDisplayLanguage("de-CH", "Deutsch"); } protected override void AddClasses() { var company = Elements.Classes.Company = Builder.AddClassWithDefaultPrimaryKey("Company", /*Guid*/, /*Guid*/); company.Caption.SetValue(Elements.English, "Company"); company.Caption.SetValue(Elements.German, "Firma"); company.PluralCaption.SetValue(Elements.English, "Companies"); company.PluralCaption.SetValue(Elements.German, "Firmen"); var person = Elements.Classes.Person = Builder.AddClassWithDefaultPrimaryKey("Person", /*Guid*/, /*Guid*/); Builder.AddInvisibleProperty(person, "CompanyId", MetaType.Key, true, /*Guid*/); person.Caption.SetValue(Elements.English, "Person"); person.Caption.SetValue(Elements.German, "Person"); person.PluralCaption.SetValue(Elements.English, "People"); person.PluralCaption.SetValue(Elements.German, "Personen"); } } Note that I created a local variable for both company and person. I did this for two reasons: * The code is shorter and easier to read * There are fewer references to the Elements.Classes.Person and Elements.Classes.Company properties. It's useful to keep the number of references to a minimum in order to make searching for usages with a tool like ReSharper of maximum benefit. Otherwise, there's a lot of noise to signal and you'll get hundreds of references when there are only actually a few dozen "real" references. [Step six: using private methods] You can see that the metadata-generation code is still manageable, but it's growing. Once we've filled out all of the properties, relations, translations, layouts and view aspects for the person and company classes, we'll have a file that's several hundred lines long. A file of that size is still manageable and, since we have methods, it's eminently navigable with a file-structure browser. If we don't mind keeping -- or we'd rather keep -- everything in one file, we can see more structure by splitting the code into more methods. This is really easy to do because we're using the elements to reference other parts of metadata instead of local variables. For example, let's move the class initialization code for the person and company entities to separate methods (unaffected methods are left off for brevity). public class DemoCoreGenerator : DependentMetadataGeneratorBase { protected override void AddClasses() { AddCompany(); AddPerson(); } private void AddCompany() { var company = Elements.Classes.Company = Builder.AddClassWithDefaultPrimaryKey("Company", /*Guid*/, /*Guid*/); company.Caption.SetValue(Elements.English, "Company"); company.Caption.SetValue(Elements.German, "Firma"); company.PluralCaption.SetValue(Elements.English, "Companies"); company.PluralCaption.SetValue(Elements.German, "Firmen"); } private void AddPerson() { var person = Elements.Classes.Person = Builder.AddClassWithDefaultPrimaryKey("Person", /*Guid*/, /*Guid*/); Builder.AddInvisibleProperty(person, "CompanyId", MetaType.Key, true, /*Guid*/); person.Caption.SetValue(Elements.English, "Person"); person.Caption.SetValue(Elements.German, "Person"); person.PluralCaption.SetValue(Elements.English, "People"); person.PluralCaption.SetValue(Elements.German, "Personen"); } } [Step seven: using multiple generators] While this is a good technique for small models -- with anywhere up to five entities -- most models are larger and include entities with sizable metadata definitions. Another thing to consider is that, when working with larger teams, it's often best to keep a central item like the metadata definition as modular as possible. To scale the pattern up for larger models, we can move code for larger entity definitions into separate generators. As soon as we move an entity to its own generator, we're faced with the question of where we should create paths for that entity. A path doesn't really belong to one class or another; in which generate should it go? Well, we thought about that and came to the conclusion that the pattern should be to just create a separate generator for all paths in the model (or multiple path-only generators if you have a larger model). That is, when a model gets a bit larger, it should include the following generators (using the name "Demo" from the examples above): * DemoCoreGenerator * DemoPathGenerator * DemoCompanyGenerator * DemoPersonGenerator The DemoCoreGenerator will create metadata and assign elements like the display languages. It's also recommended to define base types like enumerations and very simple classes [4] in the core as well. Obviously, as the model grows, the core generator may also get larger. This isn't a problem: just split the contents logically into multiple generators. For the purposes of this example, though, we only have a single core and a single path generator and two entity generators. Since these generators will all be dependent on the model's builder and elements, the first step is to define a base class that will be used by the other generators. internal class DemoDependentGenerator : DependentMetadataGeneratorBase { } public class DemoCoreGenerator : DemoDependentGenerator { protected override void AddCoreElements() { Elements.English = Builder.AddDisplayLanguage("en-US", "English"); Elements.German = Builder.AddDisplayLanguage("de-CH", "Deutsch"); } } public class DemoPathGenerator : DemoDependentGenerator { protected override void AddPaths() { Elements.Paths.CompanyPersonPath = Builder.AddOneToManyPath( Elements.Classes.Company, "Id", Elements.Classes.Person, "CompanyId", /*Guid*/, /*Guid*/ ); } } public class DemoCompanyGenerator : DemoDependentGenerator { protected override void AddClasses() { var company = Elements.Classes.Company = Builder.AddClassWithDefaultPrimaryKey("Company", /*Guid*/, /*Guid*/); company.Caption.SetValue(Elements.English, "Company"); company.Caption.SetValue(Elements.German, "Firma"); company.PluralCaption.SetValue(Elements.English, "Companies"); company.PluralCaption.SetValue(Elements.German, "Firmen"); } protected override void AddProperties() { Builder.AddRelation(Elements.Classes.Person, "Company", "", Elements.Paths.CompanyPersonPath); } } public class DemoPersonGenerator : DemoDependentGenerator { protected override void AddClasses() { var person = Elements.Classes.Person = Builder.AddClassWithDefaultPrimaryKey("Person", /*Guid*/, /*Guid*/); Builder.AddInvisibleProperty(person, "CompanyId", MetaType.Key, true, /*Guid*/); person.Caption.SetValue(Elements.English, "Person"); person.Caption.SetValue(Elements.German, "Person"); person.PluralCaption.SetValue(Elements.English, "People"); person.PluralCaption.SetValue(Elements.German, "Personen"); } protected override void AddProperties() { Builder.AddRelation(Elements.Classes.Company, "People", "", Elements.Paths.CompanyPersonPath); } } MetaBuilderBasedModelGeneratorBase { protected override void AddMetadata() { Builder.CreateModel("Demo", /*Guid*/); Builder.CreateMainModule("Encodo.Quino"); Builder.Include(); Builder.Include(); Builder.Include(); Builder.Include(); } } You'll note that we only moved code around and didn't have to change any implementation or add any new elements or anything that might introduce subtle errors in the metadata. Please note, the classes are all shown in a single code block above, but the pattern dictates that each class should be in its own file. [Step eight: integrating external modules] So far, we've only worked with generators that are dependent on the model generator. How do we access information -- and elements -- generated in other modules? For example, let's include the security module and change a translation for a caption. public class DemoModelElements { public DemoModelElements() { Classes = new DemoModelClasses(); Paths = new DemoModelPaths(); } public ILanguage English { get; set; } public ILanguage German { get; set; } public SecurityModuleElements Security { get; set; } public DemoModelClasses Classes { get; private set; } public DemoModelPaths Paths { get; private set; } } public class DemoCoreGenerator : DemoDependentGenerator { protected override void AddCoreElements() { Elements.English = Builder.AddDisplayLanguage("en-US", "English"); Elements.German = Builder.AddDisplayLanguage("de-CH", "Deutsch"); Elements.Security = Builder.Include().Elements; } protected override void AddProperties() { Elements.Security.Classes.User.Caption.SetValue(Elements.German, "Benutzer"); } } This approach works well with any module that has adhered to the pattern and exposes its elements in a standardized way. [5] In this case, the core module includes the security module and retains a reference to its elements. Any code that uses the core module will now have access not only to the core elements but also to the security elements, as well. Another major benefit to using this pattern is that the resulting code is quite self-explanatory: it's no mystery to what the Elements.Security.Classes.User.Caption is referring. [One last thing: folder structure] The previous pattern had a single monolithic file. The new pattern increases the number of files -- possibly by quite a lot. It's recommended to put these new files into the following structure: [-] Models [+] Aspects [+] Elements [+] Generators The "Aspects" folder isn't new to this pattern, but it's worth mentioning that any model-specific aspects should go into a separate folder. That's all for now. Happy modeling! -------------------------------------------------------------------------------- [1] Naturally, the IMetaModel is always available and any part of the generation process can access metadata in the model at any time. However, the API for the model is quite generic and requires knowledge of the unique identifier or index for a piece of metadata. [1] By introspection, we mean that if metadata is accessed through .NET code structures -- like properties or constants -- we should be able to find all usages of a particular metadata element without resorting to a "find in files" for a particular string. [1] It doesn't have to go there. The DemoCoreGenerator could also set up the builder (since it's using the same builder object). To do that, you'd override AddCoreElements() and set up the model there. However, it's clearer to keep it in the generator that actually owns the builder that is being configured. [1] Simple classes generally have few extra properties and no layouts or short description classes. [1] Through the IElementProvider mentioned above ]]> https://www.earthli.com/news/view_article.php?id=2718 2012-11-21T23:08:51+01:00 https://earthli.com/users/marco

"Second" <#>"Third" <#> Suppose we have a box with three links in it, as shown to the right. [Setting the height] The first step is to make this box taller, so the logical thing to do is to set the height. We'll have to pick a value, so set height: 40px on the gray box. "First" <#>"Second" <#>"Third" <#> [Aligning vertically] This isn't exactly what we want, though; we'd rather have the vertical space equally distributed. Also, if you hover over the links, you can see that the space below the text is not active. Maybe we can try to add vertical-align: middle to align the content. "First" <#>"Second" <#>"Third" <#> Unfortunately, this doesn't have the desired effect. The vertical-align property works when used this way in table cells, but otherwise has no effect for block elements. Knowing that, we can set display: table-cell for the gray box. "First" <#>"Second" <#>"Third" <#> And now the box has become longer, because the 50% width of the box is calculated differently for table cells than for regular boxes (especially when a table cell is found outside of a table). [Relative positioning] Let's abandon the vertical-alignment approach and try using positioning instead. Set position: relative and top: 25% to center the links vertically. "First" <#>"Second" <#>"Third" <#> Now that looks much better, but the space above and below the links is still not active. Perhaps we can use the height trick again, to make the individual links taller as well. So we set height: 100% on each of the links. "First" <#>"Second" <#>"Third" <#> We didn't get the expected result, but we should have expected that: the links are inline elements and can only have a height set if we set display: inline-block on each link as well. We use inline-block rather than block so that the links stay on the same line. "First" <#>"Second" <#>"Third" <#> The links are now the right size, but they stick out below the gray box, which isn't what we wanted at all. We're kind of out of ideas with this approach, but there is another way we can get the desired effect. [Positive padding and negative margins] Let's start with the original gray box and, instead of choosing a random height as we did above -- 40px -- let's set padding: 8px on the gray box to make room above and below the links. "First" <#>"Second" <#>"Third" <#> With just one CSS style, we've already got the links nicely aligned and, as an added benefit, this technique scales even if the font size is changed. The 8-pixel padding is preserved regardless of how large the font gets. [1] "First" <#>"Second" <#>"Third" <#> This approach seems promising, but the links are still not tall enough. The naive approach of setting height: 100% on the links probably won't work as expected, but let's try it anyway. "First" <#>"Second" <#>"Third" <#> It looks like the links were already 100% of the height of the container; in hindsight it's obvious, since the height of the gray box is determined by the height of the links. The 100% height refers to the client area of the gray box, which doesn't include the padding. We'd actually like the links to have padding above and below just as the gray box has. As we saw above, the links will only honor the padding if they also have display: inline-block, so let's set that in addition to padding: 8px. "First" <#>"Second" <#>"Third" <#> We're almost there. The only thing remaining is to make the vertical padding of the links overlap with the vertical padding of the gray box. We can do this by using a negative vertical margin, setting margin: -8px. "First" <#>"Second" <#>"Third" <#> We finally have the result we wanted. The links are now large enough for the average finger to strike without trying too hard. Welcome to the CSS-enabled touch-friendly world of web design. The code for the final example is shown below, with the sizing/positioning styles highlighted: .gray-box { background-color: gray; border: 1px solid black; border-width: 1px 0; width: 50%; text-align: center; padding: 8px 0; } .gray-box a { background-color: #8F8F8F; display: inline-block; padding: 8px 20px; margin: -8px 0; }

First Second Third

-------------------------------------------------------------------------------- [1] Naturally, we could also use .8em instead and then the padding will scale with the font size. This would work just as well with the height. Let's pretend that we're working with a specification that requires an 8-pixel padding instead of a flexible one. ]]> https://www.earthli.com/news/view_article.php?id=2605 2012-01-08T17:13:18+01:00 https://earthli.com/users/marco

illustrates quite nicely how programmers approach the world of problem-solving. [image] The chart does not show just much time must be spent before the programmer wins, that being dependent on the complexity of the task. The probability... ]]>

illustrates quite nicely how programmers approach the world of problem-solving. [image] The chart does not show just much time must be spent before the programmer wins, that being dependent on the complexity of the task. The probability that the task will recur is also highly relevant, as automation of a smallish, one-time task is useless. Neither of those things will stop a determined programmer, though, who will automate no matter what. ]]>