DCC++ Update Project 2020

FlightRisk Feb 16, 2020

  1. Atani

    Atani TrainBoard Member

    1,460
    1,697
    36
    I've submitted a few patches to JMRI previously for DCC++ fixes. I can submit new ones as well if needed.

    The last time I looked there was limited support for this, but we may be able to add some custom support to it. I did add the ESP32 response parsing to it previously.

    I had thought of this as well but never implemented it. The only problem I have with including everything out of box is about the binary size, at least on the ESP32 there is plenty of space but even there I have hit ~95% usage by enabling certain sets of options and it is risky to have it that full.
     
  2. FlightRisk

    FlightRisk TrainBoard Member

    548
    237
    14
    Yes, we can coordinate with them on the development hub and also on the JMRI forum to do exactly this. I was talking with them about changes to the way current is reported from DCC++ to JMRI with the <c> command. They are very open, especially when they see something spec'd out. For what I looked at, I could even do the pull request myself, but they are willing to takes something from us and them implement it too.

    I am pretty sure ESP32 does this. I asked Atani about this here or another thread. The change on our end with the code already there is trivial. If you have a I2C display attached (something in the code we haven't talked about yet) it would be a nice feature even without modifying JMRI since that could provide a response verification and status.
     
    Last edited: Feb 26, 2020
  3. John W Zerbe

    John W Zerbe TrainBoard Member

    96
    59
    7
    Well, the Uno only has 32k of flash memory where the Mega has 256k. Tiny by today's standards. Not enough for all the options we've discussed, but possibly enough to give people the option of two or three motor shields in the entry level base station that has been discussed early in the thread. I'm just looking for a way to make its use as simple as possible for first timers.
     
  4. RoadRailer

    RoadRailer TrainBoard Member

    41
    11
    4
    Release plans can also play a factor in branching strategy. For a more frequent, rolling release strategy with incremental updates, a dev branch can end up being extra overhead. This approach is somewhat analogous to the DevOps-style approach documented at the link below.
    * https://docs.microsoft.com/en-us/azure/devops/learn/devops-at-microsoft/use-git-microsoft

    Implicit with this approach is that the feature/topic/bugfix branches each represent a "minimum viable product" once completed and merged to master; additionally, unless there is a need to create a "hotfix" for an prior release (instead of rolling forward with the current release), release branches don't see much use.

    As I would assume this is a side project for most of us, just a suggestion to minimize the overhead related to maintaining branches.... As @Atani referenced, though, just so everyone is on the same page.


    Related to the discussion of C# and WPF, if going a .NET-related route, I would suggest consideration of UWP / Xamarin / platform.uno as a front-end route to better facilitate cross-platform portability capabilities.
     
  5. FlightRisk

    FlightRisk TrainBoard Member

    548
    237
    14
    @Atani , @John W Zerbe , @Keith Ledbetter , et. al. Let's go ahead and decide on our structure and "get a move on" as my father liked to say ;) Based on our discussions, and unless there is any dissent, let's implement the following:

    • Use the Gitflow workflow (described HERE for you lurkers)
    • master is the current stable release candidate
    • develop is where we push the changes
    • Have features branches that will use feature/git-issue-id
    • Have hotfix branches that will use hotfix/git-issue-id
    • Branch features from develop and merge them back to develop for consideration to go to master
    • Release branches will be labeled release/x.y
    • All commits to master will be tagged with x.y.z (major, minor, point)
    • Point releases fix an existing release, minor version indicates feature additions, and major versions are significant adds and/or breaking changes
    • Setup Github actions to automate the test build process for PRs
    • rename DCCpp_UNO fork to simply DCCpp to reflect software that now runs on at least 3 different Arduinos
    • create a DCCpp Classic repo to contain the final and only touched to fix a bug release of the original DCCpp_Uno from Gregg
    • handle @Atani's idea restructuring the code to clarify the fact that the esp8266 Wifi (for the esp8266) and BaseStation (for the AVR) are completely different projects
     
    John W Zerbe and Atani like this.
  6. FlightRisk

    FlightRisk TrainBoard Member

    548
    237
    14
    Are you saying take out 2 steps? I had initially questioned if we need a develop branch, and release versions can just be archival. Do we want to revise my previous post?
     
  7. RoadRailer

    RoadRailer TrainBoard Member

    41
    11
    4
    Yes, if we're on the same page,
    * There would not be an intermediate develop branch between the feature/hotfix branches and master
    * Changes are validated against master during the pull request process; if the changes doesn't pass validation, then no harm, as the reverse integration merge has not yet been committed to master.
    * Release branches could essentially be considered archival


    Regarding the starting point for a revived coding effort, at one point there had been discussion of building off the library approach of the Locoduino project ([Website](http://www.locoduino.org/) | [GitHub](https://github.com/Locoduino/). Is that something still open to consideration? Particularly for an open source project, having a broader base of interest and collaboration can help reach that "critical mass" that is necessary to sustaining a project's viability over the longer term (e.g. to potentially avoid the fate of the original DCC++ project).

    From member @Trusty :
    Beyond the open source and collaborative aspects, usage or implementation of a library approach would seem to be a good software design practice to follow anyway.
     
    Last edited: Feb 27, 2020
  8. FlightRisk

    FlightRisk TrainBoard Member

    548
    237
    14
    It gets pretty complicated. The library approach (which I am familiar with as my commercial software is a DLL and an ActiveX) is a good one, but I see it as a completely different product. The "language" and flow is a different learning curve. Granted, that layer of abstraction makes the code smaller and more like writing a macro, but it is a separate path. We can certainly keep talking about how that fits in. I know the author (@Trusty) evangelizes it here on occasion, as youv'e seen ,and is more than happy to help. Maybe he will provide more input.
     
  9. FlightRisk

    FlightRisk TrainBoard Member

    548
    237
    14
    *** Team *** (all coders) Final vote on workflow:

    Everything I said in my previous bullet list except remove a step:

    Preference? Any reason to NOT use this slightly simpler method?
     
  10. Keith Ledbetter

    Keith Ledbetter TrainBoard Member

    279
    195
    12
    I like the simplified version and agree.

    On the library. I guess my fundamental question is do we feel a library approach is the best or is the more open code is best. I think and again my limited knowledge will show through here that libraries are more difficult to maintain and my fear is if we purely rely on @Trusty its again the single point of failure we're trying to avoid.

    Having said that if we think the library approach is technically the best and we want to head that route we could clone @Trusty version here and use that as our base.

    Again I am not knowledgeable enough to make that call so others like @Atani, @FlightRisk , @RoadRailer etc should have the say in something this fundamental.

    From a user perspective a library seems like a very good choice but I get that then someone like me can't just go in and edit stuff like I can now and like I say if the knowledge base of how they work and best to manipulate/change is not in this group then wouldnt make sense.
     
    bocabob likes this.
  11. John W Zerbe

    John W Zerbe TrainBoard Member

    96
    59
    7
    Well, from my perspective on the @Trusty 's Library approach:
    Pros:
    1) for beginners, the black box aspect of using the library apis is attractive.
    2) for someone wanting to extend dcc++ with their custom extensions that they don't intend to share, it gets up up and running very quickly.

    Cons:
    1) given the number of features discussed on this thread and others, I suspect that the library will quickly grow in size such that it won't fit in arduinos flash memory. this could be mitigated by very careful design of multiple libraries where you pull in only the libraries of the features you wish to use and have, for example, a motor shield handler library per hardware device that "registers" with the main library on initialization.
    2) I don't know much about the arduino library management workflow, but I can't imagine that it would be something we would want to manage for possibly a dozen or more libraries ultimately.

    On the topic of the development branch, I still would like to keep it as the first "feature integration" branch prior to merging to "master" were we keep what we THINK is going to be the next stable release. "alpha testers" would be using/testing the development branch which will have more rapid changes than master. This is my vote but am ok with working with the group on what the majority comes up with.
     
  12. FlightRisk

    FlightRisk TrainBoard Member

    548
    237
    14
    Workflow - @Atani , your vote? Maybe one other coder? I like the structure of the dev branch, but the simplicity of pulls to master. I don't *think* this project with have a huge flow of PRs, but hard to tell.

    Let's take the library version for now. If people want that, the solution is on Locoduino. If you want to play with it and see something particular and are passionate enough to "sell" the idea, we can revisit.

    And I wanted to share with the group that Trainboard has just awarded me a "trophy". Please try not to be envious
     
  13. Sumner

    Sumner TrainBoard Member

    2,798
    5,837
    63
    Just a note. I don't have figures to back this up but I'd imagine that the majority of potentia newl users of DCC++, myself included, are most interested in a stable version of DCC++ that lets them work with JMRI to program decoders and run trains and are not interest in if they have a mega or uno or which motor shield they use. Give them clear instructions on, buy this Arduino and shield, do this to it, load this version sketch and connect things this way and you will have a reliable system to run your trains with.

    Options to have greater control on the layout can come later for them and they can then decide if they want to pursue them. An installer to load the sketch would be the biggest help to them.

    I think what Les and Thomas Davis posted here pretty well summarizes what is keeping more non-computer people from trying the DCC++ approach...

    https://www.trainboard.com/highball...-money-on-this-dcc-setup.130188/#post-1125222

    Don't get me wrong, I think it is great you guys are working on DCC++ and fixing bugs and expanding its capabilities as that is the future for it. Right now though it would be nice to just have Gregg's version in a stable condition and a simpler way for a new user to load it and implement it with JMRI. A first step maybe? I'm using it and haven't run into any problems to this point, but my needs are minimal. Is it good to go the way it is?

    Thanks for all that you guys are doing,

    Sumner
     
    John W Zerbe likes this.
  14. RoadRailer

    RoadRailer TrainBoard Member

    41
    11
    4
    I am also hoping that @Trusty might chime in here, but I would note the following:
    1. Locodunio is setup on GitHub as an organization (i.e. not under a particular individual's profile, such as @Trusty's own https://github.com/Trusty77)
    2. The Locoduino organization currently has multiple members (3)
    Perhaps this DCC++ implementation project could be added as another project within Locoduino, but it would perhaps be best for someone on the Locoduino project would need to weigh in on that.

    The library is itself open source, so it wouldn't be like coupling this effort to some "closed box" functionality; worst case, this project would perhaps end up fully maintaining the library (as opposed to sharing responsibility, whether via creating pull requests against the upstream repository or as outright members).

    Also, it might help to clarify that this is not advocating for a single, large, all-encompassing library.

    In the longer term, approach to software that is tiered and comprised of multiple libraries can be easier to maintain, as opposed to an intermingling of various responsibility layers. If there are changes, a tiered approach can also help towards scoping what might change/break. Maintainability is a primary motivation for a tiered approach; potential pick-and-choose by the end user is secondary and (as long as there

    Along those lines, I would concur with @John W Zerbe 's observation above that a library approach might actually make it easier to jump in and edit things, as there would (hopefully) be a clearer delineation of what code does what.

    Example: Implementation of lower-level details such as DCC waveform generation is closely bound to the NMRA spec, so as long as the NMRA spec remains stable, "casual" developers shouldn't need to be concerned with such details. Abstracting away such implementation details can thus make it easier for a casual developer to tweak things without risking breaking core functionality.

    Application: @Atani highly advised migrating the interface to LCC, which (from all I've seen) is very solid advice. Regardless of what interface type(s) are supported, though, the core DCC standard itself is not changing, so (to the extent to which that implementation is correct) the waveform generation logic likely does not require much (if any) change. However, if interface implementation code is intermingled with DCC waveform generation code, it can be harder to update interface support without breaking DCC waveform generation logic. Thus, by leveraging a DCC waveform generation library to implement a given interface, there can be some built-in protection against breaking DCC waveform generation logic.

    For fuller DCC++ library details from the Locoduino perspective, I would suggest reading this article on why a DCC++ library.


    A DevOps process manages release integrity through the pull-request workflow. A pull request can be built and evaluated against the target branch without requiring the code to be fully merged and committed/checked in.

    With an intermediate/pre-master branch, feature distinction is potentially lost once it is merged in with other features the intermediate branch. If it turns out that a given feature is not quite ready, either everything is held up until that one feature is fixed, or the changes to exclude have to be "cherry picked" around.

    With a pull request workflow, if a feature isn't ready, then the pull request is not merged/committed/checked in (preserving the integrity of master) and the branch remains (maintaining feature branch distinction).


    What scope is envisioned for DCC++? Is there a point at which these additional features start overlapping with ESP32 CS? A few quotes below for contemplating the scope question:

     
    Keith Ledbetter likes this.
  15. Atani

    Atani TrainBoard Member

    1,460
    1,697
    36
    This is by far the biggest issue with maintaining the development branch and is precisely why ESP32 CS v1.5.0 has not come out yet. The change set on the intermediate branch grows to a point where it is a major challenge to test and ensure limited breakages. Once I complete my current testing/migration work on my development branch I will be dropping it in favor of the feature branch approach instead with the feature branches only merging down to master once things are ready to go.

    I don't think that is the right direction for this project. However, using the library to *ONLY* generate the DCC signal and possibly current detection aspects might be doable. The library was based in part on the DCC++ project so the integration points should not be too challenging. One other problem with libraries, versions.. End users will pick a version (or use an existing that they already have) and it can lead to unnecessary failures due to unexpected versions being picked. PlatformIO allows specifying library version requirements so it is not a major problem there but Arduino IDE has no such support.

    Now, another option that would work is to break the current DCCpp_Uno tree into a handful of smaller projects, similar to the work I'm doing on ESP32 CS (ref) but again, this will not work for Arduino IDE.

    I think you missed a word or two in there, but I think you are intending to *NOT* pick up the library model for now since Locoduino provides a CS example with their library today?

    I think your proposal (no dev branch) is fine. The release branch can be omitted in favor of a git tag to mark the point in time when the release was made (GitHub web UI does this when you create a release). The only piece missing would be the GitHub Action integration which we should get in place ASAP so that all PRs will pass through validations prior to merge, at minimum a sanity test compilation. I can provide a starting point for testing a PIO based build and we can also test with Arduino IDE but it will take a tiny bit more work.

    This needs to be the top priority, provide a 100% stable and reliable base for people to start with. For people that need a more advanced system (multiple motor shields, different motor shield, etc) we should provide clear instructions on how they can accomplish that.
     
    Mani and Sumner like this.
  16. bocabob

    bocabob TrainBoard Member

    47
    22
    6
    I’ve been lurking and thought I’d provide my perspective (for what it’s worth). My demographics are:
    · Recently returned to the hobby
    · 22 year old Digitrax system in place using JMRI and Engine Driver app
    · Looking to modernize and animate
    · Computer literate (decades in H/W and S/W development)
    · Moderate coding skills (haven’t been professional for 35 years but I keep a hand in)

    I won’t comment on your workflow, just adoption and use case.

    When I started looking at DCC++ in 2018 the first thing I noted was that it was not being updated and Gregg had gone away. That was a non-starter for me. But there were forks and Locoduino so I kept reading. The four things important to me were, in order:
    1. Reliability and Support
    2. Function
    3. Ease of implementation
    4. Extensibility

    I was very tempted to go with Locoduino but in the end decided on Atani’s ESP32 CS. While I expect to stay on that, I am interested in this project as I see its success strengthening the open command station market and perhaps a hook for some young tinkerers to get into the hobby. I also think it would make a great program for my NMRA division meeting as it does come up from time to time.

    I think the discussion has been hitting all the right points on positioning, usage and adoption. This should be the starting point for most people wanting to use DCC++. Keep it simple, keep it cheap. People can try it and grow from there. I do think the JMRI connection is critical and that should be bullet proof.

    Remember that for most people this is a means to an end, they just want to run trains.

    I agree with all the other points made. Thank you all for taking this on.
     
    Sumner, Mani and Atani like this.
  17. RoadRailer

    RoadRailer TrainBoard Member

    41
    11
    4
    Grammar questions aside, perhaps this also goes back at least in part to ensuring that there is a common understanding of and agreement on scope, including what might be the uniquely distinguishing characteristics of this project compared to other projects? On the one hand, there have been comments about potentially being limited by capabilities of the Uno and starting with the Mega so that there is room to grow; on the other hand, there have been comments about essentially bare-bones DCC signal generation without command station capability logic.

    Based on quotes such as those below, I'm not sure I've heard a collective agreement here yet.

    (emphasis mine)


    So, in short, what is to distinguish this project from existing projects, including existing projects such as Locoduino and ESP32 CS?

    I concur that is key:
     
    Last edited: Feb 27, 2020
  18. Atani

    Atani TrainBoard Member

    1,460
    1,697
    36
    I would agree, the scope of the project has not been fully agreed upon nor has a roadmap for possible expansion (features).

    As you have rightfully pointed out:
    This needs to be defined. Each of the two projects you have mentioned have defined audiences and feature sets. This project very likely needs to be very narrow scoped (to start!) and make sure that remains stable and easily deployed for the end users. After that has been accomplished it can be built upon for other aspects and NMRA compliance issues (there are a few possibilities).
     
    Sumner and RoadRailer like this.
  19. bocabob

    bocabob TrainBoard Member

    47
    22
    6
    My prior post was based on the feeling that I got that this thread was dancing around the concept of the project being a fixed and supported DCC++ implementation entry point with a growth path to enhanced feature sets and/or jumps to other projects of a higher order. As you decide on your scope, I think you need to map out this positioning. The good news is that since the H/W is so cheap it is really not a deterrent to hopping up a level to support advanced features once they become desired.
     
    Atani likes this.
  20. Keith Ledbetter

    Keith Ledbetter TrainBoard Member

    279
    195
    12
    In my humble opinion here is what I think it should be:

    1) First and foremost a quick start that has the original base code with whatever small tweaks make sense to be NMRA compliant, etc. and ultimately awesome if @Dex or others could actually pull off an installer. Truly the only difference in this and the original is tweaks such as the read and write packets we have briefly discussed and the installer.

    2) A second fork that is the latest and greatest DCC++ as decided by the org. The trick to me is what's the base for this but lets assume we agree the base for latest and greatest. It should then hopefully be continually developed and documented with things such as hopefully an ever growing number of motor shields that are tested and proven to work(or document if they don't), support for LCC, wireless, bluetooth, etc multiple power district support,etc. I am perfectly fine with that being Mega based only given there is no real advantage to the UNO other than a couple bucks and a slightly smaller form factor.

    3) A document holder and side branch for related things like dcc++ wireless throttles

    At some point I might expect that in order to do what I'm discussing above in #2 we may indeed not be able to support everything given the Megas limitations but I think we should get there first then worry about that.

    The real answer may very well indeed be to do my #1 above as this project and then help @Atani out getting the ESP project as the next generation bigger and better with all the support things mentioned in#2 but I'd like to try to max out what can be done with the Mega (and therefore DCC++) given how cheap and easy it is for everyone but again I'm one vote here and will help and support as much as I can whatever the collective decides. I do think we should not confuse DCC++ with the ESP work @Atani is doing. Totally different command station hardware and software. If we go that route I'm quite sure we could get JMRI guys on board for providing support for this new command station but it's not DCC++ as he has said.

    I don't disagree locduino is not that far off from what we are discussing but the language barrier, more focus on the hardware, etc while not insurmountable with google translate makes it tough.
     
    Last edited: Feb 28, 2020

Share This Page