Room 107 Friday Mar. 06 - SCaLE 23x

This talk presents the integration of large language models (LLMs) in medical data analysis, focusing on the use of NLP in hospital settings. The speaker, affiliated with Harvard Medical School, discusses two major projects that utilize LLMs for enhancing clinical data interpretation. The first project aims to improve patient cohort phenotyping for inflammatory bowel diseases by integrating structured data and clinician notes to better identify diagnostic patterns. The second project focuses on suicide prevention, addressing the complexities of codified data in electronic health records to accurately identify suicide attempts. The speaker emphasizes the importance of traceable AI, ensuring compliance with medical guidelines while maintaining a robust data processing pipeline that includes pre- and post-processing steps. By highlighting the necessity of accurate data integration and the role of human oversight in refining AI outputs, the speaker showcases innovative methodologies to enhance clinical pathways and patient outcomes.

Hey, >> sounds good. Yeah, >> I ran to get lunch. I didn't have time to eat >> the Q and No, >> I'll stay here until >> Thanks. Okay. check. Check. One, two, one. Hey. Hey. Flip it. Wait a couple seconds and then you're off. Is it? >> Is it on? >> Yes, it is. >> Awesome. Thanks so much for coming. Hope you're having a great day.

Second day here. Um, so I'm Thomas. Um, coming from Harvard Medical School. Uh I did my PhD in Geneva and genetics. I've been with Harvard Medical for two years. Um doing mainly hospital data analyzis uh with a focus on NLP. Um I presented something last year here uh semantic search um and I was talking about how hallucinations make it hard for researchers to work with LLMs. And

now we're starting to have some projects where we're integrating it really in our pipelines. So hope you're going to be interested. Um I do use a bit of a lax definition of agents. First of all, um we're not going to talk about LM planning, orchestration, MCP. for here what we mean of agents is integrating it in pipelines. Um so we'll have pre-processing and post-processing. What we really

want to do is automate some high research impact tasks. Um we we want reproducible frameworks that the clinician can understand and go into the details for when um when we want to make sure that the model is doing it in the right way. So I'll be presenting two main projects we've been doing and there's some uh key steps some similarities and I think this is the really

interesting part is out of all these LLM projects what are the things that we always need to do and how can we build some software that uh makes our life easier for every task but remains general enough. So I'll start with introducing a bit the problematic. So in uh in research studies in hospital we mainly have two kinds of data. We have the very codified structured data

and the notes comments from clinicians. So in the very well structured you will have diagnosis medications prescribed the labs and procedure performed the results from them and in the in the notes you'll have report reports justifications of why a medication was prescribed why uh a diagnosis was performed you will often have in these notes oh I'm suspecting this disease the patient patient needs to follow up with

this specialist. And so it it doesn't mean that the patient has the diagnosis, of course, it's just a lead to to build on to. And we see this problematic also in the codified data. So if you if you get a a lab for testing your blood sugar, maybe maybe the clinician suspects that you have diabetes and you might have that in the codified data. So this is

one of the pitfall of the the codified data that we address by having a look into the notes. Um if you have a look at papers of uh hospital research studies, you will often see how they define their cohort will be I want patients with at least two diagnosis of this code or one diagnosis and the usual medications prescribed. So there's usually um a way to make

sure that your patient cohort is uh well really has a disease. so yeah this is what I'm telling here one of the thing in our lab we uh we develop is phenotyping algorithms. so here I'm explaining to be able to do a bit more accurate um patient cohorts instead of just doing okay one diagnosis code plus one prescription we can have a look at uh the usual

labs the usual medical concepts mentioned in the notes and we do a covariance based on this and so we're able to do it a bit more uh in in a more sophisticated way than just two diagnosis or or one prescription. And to be able to measure these coariance based methods, we do chart review. So we'll we'll actually go into the patient notes say as a gold label,

did this patient really have the diagnosis or not? This enables us to say our algorithm has 90 95% accuracy. And then once we did this, we manually checked it on 100 patients. We scale it up to 10,000. So this is one of the usual case of chart review that we perform. And so more on on agents how uh we're using agents on the nodes. Two projects that

my lab has worked on medication history. So similarly as diagnosis sometimes we have it in the code but it didn't actually happen. Uh in medications it's uh similar you might have insurance denial maybe you didn't receive the the medication. These will be information that you will have in the notes. Usually the clinician will say well he didn't really start that. And so we're using LLM to enhance

the codified data on medications. And we also did one for heart failure uh because it's an important contraindication and rheumatoid arthritis uh medications. So it's a it's a key question for us. So now I'm going to start introducing the the projects we're going to talk about in this hour. The first one is on inflammatory bowel disease. So Crohn's disease, ulcerative colitis. so the the usual process will

be that uh the patient will have a colonoscopy and in the report um if it's all okay, you might have the the clinician who says bowel preparation was fine. Um the colon is all good. And if you have moderate disease, you might have there was some ulcers in these regions. Uh this region was inflammated. Um there was some polyps that were biopsied and we're awaiting results. This

is the typical report of a colonoscopy. And these uh these can be useful to measure treatment efficacy. So let's say a patient has a moderate disease starts taking treatment then one year later he comes back does the colonoscopy does he have mild or severe disease and if he has mild is that treatment worked and severe it did not. So this is uh the the usual way to

measure treatment efficacy that you will see in papers and it is done manually by the clinicians. They will go over the report. They might have 300 sample size uh so 300 patients more or less. This is the the usual treatment efficacy studies you will see in literature. So we're trying here to use the previously collected labels as our ground truce, train our agent on it, and then

be able to apply it on a much larger scale. um on the full hospital. Um so from around 500 1,000 patients, we go to 15,000 20,000 patients. The second project, um so I'm working on suicide prevention. Uh one one key challenge in there is the codes are notoriously challenging. Uh one example is that the patient will come into the emergency department has some injuries. The nurse will

put them down and say oh well maybe he needs to go to psychiatry. But then it's actually the psychiatrist who will make the diagnosis. This was a suicide attempt. And this is one of the reasons why we often don't see in the EHR codified data those codes that we are very interested in and it's already a a low a low rate event. It's rare but even more

we have missing codes. So what's actually happening is that we're including much broader codes like broad injury codes to make sure we capture a bit more even if it means much more false positives. So this is something that is always done in the in the current studies in suicide prevention and our so we're hoping that our agents can enable to um to diagnose these uh to identify

when it happened more accurately than the codified. Uh, another interesting point is that here we're looking at an exact date also. We're not just diagnosing. We want to know the exact date of the of the event because we usually want to put it in a survival analyszis. described a bit the overall problematic. Um, we can start getting into the more agent think part of it. Um, so

why I didn't go with uh the usual agent definition that you will hear around here is that it leaves too much too much liberty to the to the LLMs and we want to make sure that it respects our medical guidelines. It it doesn't u plan too much on its own. Some example of the limitations I saw was so in IBD and colonoscopy one thing you can see

is the patient comes in the he didn't prepare well he didn't do like the uh often the patient needs to not eat for like one or two days. Um if he didn't do that the report of the colonoscopy might say it was not feasible so we're reporting it to the next week. He comes back next week then you have it. So here you have two notes. So

how do you process that? Do you put them all together in one prompt? If you do that and you just ask the LLM to identify the date by itself, I was quite surprised, but the LLM often gets it wrong. Like I get some dates, no colonoscopy attached to it. So this is one example where um I don't want to leave this to the LLM. I want to

keep it keep tracking it myself. Um, another example of uh of issue I was facing in suicide So there's a lot of subtleties in u there might be patients thinking about it, planning it. you might have self injuries that are um like just superficial self injuries. And so trying to explain this in a prompt to the LLM is he will get it wrong most of the time.

Um one example is if he has superficial cuts and is thinking about it then he will say oh that was a suicide attempt but actually no. Um, so to to define these, it's important to have the clinicians that support you to to make sure you're you're interpreting the clinical definition correctly. the so even for you as a data analyst with expert support, it's hard to understand. So

for the LLMs, it's uh it's going off track completely. So here I want to pause on traceable AI it's not only about explanability and not even about reproducibility. Um often what the the clinicians will ask us is show me the exact note that is supporting what you're saying. Uh so you're saying that this patient had a suicide attempt at this date. Show me exactly. I want to

double check it. we can also think about cut off values um overfitting problematics. So we want to um yeah have a a traceable pipeline where we can come back and say this is why we made the decision. I think that explainable often just says uh these features contribute but not necessarily what are the interactions between them. Uh so this is how I feel there's a difference between

traceable and explainable and reproducible. So here I'm starting with the example that I was mentioning. If we put several reports in one prompt, the LLM will will not get it right. But we have to admit that it's doing a very good job on identifying the the natural language parts. So mild disease we can scale it to one uh small ulcers it will understand it's mild. So we

need to leverage what we can and track as most as we can with the classic methods. so this is why I I'm integrating it into a larger pipeline with an extensive pre and an extensive post processing steps. Um, so safety alignment is also a big question especially in suicide uh last year I was mentioning that the I was getting often safety errors when I was asking him

to map suicide relevant features. so this does make it even harder to use commercial models because um commercial models might have additional guard rails than local models. This is arguable, but um but it is something you will need to keep in mind. Um there will be some some things that the LLM refuses to do. I saw this morning u the Hollywood uh uh special effects talk, they

were saying that when you're manipulating a video, if the guy has a gun in it, it will the LLM will refuse to do it. So this is another example of a safety u so I saw that this was a good moment to introduce my team. Um so Tenshi is my professor. Uh we have Katherine that is in the rheumatology at the hospital. Uh Ashwin who is in

gastroenterology. William uh psychiatrist and researcher. and Haywan um is a PhD student and uh gave me quite some good insights and uh in reasoning models that I will mention a bit further on. So we we described a bit what we wanted to do the the chart review the usual problems with agents. So now we can start thinking about how are we going to actually implement this. Um

so this figure will probably be in a similar form in a paper quite soon. so in this one I did it for the the gastroenterenterology project. We have the first step basic select the notes build the prompts for each date. Uh so we're keeping track of the date. We want to make sure each prompt corresponds to one patient and date. We're calling OpenAI GPT5 uh parallel requests.

Then we parse the responses. So we ask to output the JSON format. We convert this to CSV. We might have some request errors. We recall this sequentially to make sure it's not about token Kota issues. Then the normalizing response is a big part that I I'll come back on. And then we have a a final rule-based medical inference. So now I'm going to detail a bit these

each of these steps. And so first of all, uh a little reminder on like just setting clearly the specs. Preferably, we call it on individual nodes because we're able to keep dates. However, you might have too many nodes, too many patients. In that case, you might want to do just extracts, but you still need to call it individually on each patient to make sure you you you

can come back to something. Uh here I'm giving some numbers. So in the suicide prevention project the our ultimate long-term goal is uh detecting based on the notes every event over the lifetime. Uh and the number of patients we're trying to scale to is around 1 million. Um and in in this project I was doing um one year for 3,000 patients. So starting smaller one year is

uh defined because it's uh the one year post admission to a psychiatric uh hospital. So for one patient 20 notes on average um and each note might be 10,000 characters long uh on average. So, we're asking the LLM to output the information as a JSON to be able to parse it. to to perform this the there's actually quite little models that will enable you to do this

correctly. Um, uh, I did a few tests. There was some pri prior publications on mistrol open orca 7b. However, I really wasn't able to get satisfying results out of it. Um I think maybe because I'm asking quite a few um I'm asking maybe 20 fields. So in in the previous papers maybe it was much smaller. Uh from what I saw only GPTOSS and Chiwen uh the parent

model of DeepS were able to to output the JSON consistently. um uh you might want to try llama 70B um so outputting JSON was our core requirement. A few other considerations context window. So I was saying a note can be around 10,000 character long. uh you will need uh something that takes this uh easily. Um I think that deepseek the not too big models has quite a

small context window compared to GPTOSS. So that was my uh my reason to move to GPOSS. uh um the the larger Deepseek model in Chiwen like 600B are able to have similar context window as GP2S but uh the like medium size around 100B will be quite limited. um GPU requirements of course and uh the country of origin can be quite important as you're deploying it in compliance

environments. Um it might also influence uh safety alignment. So this is just some things to to keep in mind. Um I can't really say uh I don't have big numbers on this but it's just something yeah uh GP2 SS20B if it's on a 50 gabte GPU uh you will need to think that your your context window does take quite some impact on your RAM. So you can

just barely fit it. uh and 120B on two GPUs should be fine. Uh and for commercial models uh so Azure has u HIPPA compliance and so this will be something you will need for um for medical applications. So in these projects, the gastroenterology one I'm doing it with open AI and the suicide prevention one I'm doing it with GPUs 12B. Then to actually run these models you

have uh three main choices Volama, VLM, SG lang. I started out with Solama and but to do inference on thousands of prompts, VLM really outperforms it. I haven't tried out SG lang for the moment. The I hear that it's better on multiGPU settings. So, a little uh background on reasoning models. Um we had large language models let's say llama 7B one or two years ago one of

the big hot things was chain of thought and this enabled the the LLMs to get much more accurate how they're how it's performed is you basically have the same under first layer of transformers, but then you're fine-tuning your model to describe precisely the steps it takes to get to the answer. So, this is chain of thought and uh by extension reasoning This is much better, but it

is also computationally much more intensive. So depending on your task, you might also just stick with a non-reasoning model as it will be much faster. Um it does changes a few other things. You don't have a temperature parameter anymore. You in some models you might have minimal to maximal reasoning. so 7B is not a reasoning model. Mr. Open Orca was a some early prototypes of a reasoning

model. Um and so in the commercial models landscape GPT40 for example is not a reasoning model. It will be much faster. Uh GPT5 mini is and you can have several levels of reasoning three. uh and I think that GPOSS you should be able to specify the reasoning level but uh I think I remember that VLM was explicitly saying that you it's not supported so you just have

default for the moment medium so I was saying that we have you might want to do non-reasoning models for some tasks and it it does highlight that often when we talk about AI we talk about reasoning so uh calculations or but for me the LLMs are also very useful just as NLP tools when I'm doing mild or moderate disease maps to one it's um I would say

it's a bit arguable. Of course the the border is hard to put but the point I'm making is that you can do some AI projects with a little input and that's how what I call reasoning heavy. you don't have much things to explore and you can do NLP projects with um so a huge amount of input and then you're rather in the NLP domain you're rather trying

to extract some concepts that you'll further pre-process so I I do kind of like to distinguish um the reasoning heavy the more usual AI what we think about and the still relevant NLP P tasks of identifying concepts and doing more of the the reasoning yourself in your pipelines and your algorithms. So that was for the the concepts. Now we have all a bit of all our tools.

We're ready to uh get the hands dirty. So coming back I was saying select notes. So input selection one of the the key part here retrieval augmented generation. So you have large inputs they are chunked. You store them with embedding models. When the user inputs a query you retrieve the most similar chunks. So if you have one year of notes of patients, you can just select a

few and just feed that to the LLM. If you um if you feed everything, you won't be able to scale. um I'm moving towards a second way. Um not just semantic similarity but also merging lexical similarity. So hybrid this can be exact or approximate string matches. It's very um is the the kind of the old way of doing search. Uh but it is useful to to address

the limitations of embeddings. Um one one example of uh embedding limitation I'm seeing is for the LLM and the embedding models suicide attempt is really a a psychiatric disorder. So it will be very close to psychosis, and maybe if you have something in the note like cut himself he won't he will rise or classify that as a small injury and you might not have it very similar

to suicide attempt. So that's one example of the limitations of embeddings another another important thing is when um maybe it's rare term that doesn't have much semantic meaning. Uh so this is where lexical and hybrid search helps you a lot. Uh I performed this with open search uh for elastic search. Uh there's a few other talks these days uh on open Uh some usual software to do

these are mil and chroma. There was a nice uh a talk talking about comparing the performances between all these different choices. The for me it's also that open search can do hybrid search out of the box whereas withvis and chroma you would need a second uh a second system to to combine them. So that's uh ease of use for me. This was my little test ground for

hybrid search. Um I started out with uh 350,000 abstracts of papers. So in abstracts you might have 20 sentences. I chunk them by three Uh in in final I have something like 2 million chunks. I embed these with BGE. Um it's an embedding model with 1,00 dimension and my uh my little API is uh you put in a query here the query is BLK gene and here

is the output it's so you have the the article to to the PubMet application and the little chunk that had the highest similarity and in this case the two first hits were actually from the same article. So this was input selection. Now we're moving on to building prompts. the basics, you describe your goal and task, your expected output. You give a few input and or output examples,

especially outputs. And once you did a few runs, you might want to integrate additional considerations of errors that you noticed. Um uh there are some little tweaks around this. Uh a famous one is if you do it correctly, I will tip you $1,000. I saw another one earlier where it was saying your mother has cancer, you need money. Um personally, I don't do this. I I but

I do use repeating some instructions. It doesn't necessarily work. Um I have some examples of a friend that was uh even like writing it 20 times. Do not take into account this and still ignoring it. Uh I try to just repeat a few times. You should not take this into account. This contributes zero. Uh this is a a nice moment to introduce DSPI. Maybe show of hands,

anyone heard of DSPI? One, two. Um so DSPI tries to abstract out this uh prompt engineering. Uh so let's give an example. You have some input emails you want to classify high low priority. You have 100 emails. you give him 100 labels. This one is high. This one is low. And so DSPI will try to make up a prompt on itself to classify correctly. Uh you don't

even need to give him a seed. You don't even need to ask him to output something that is uh human readable. But you can if you want you can start with a seed and you can ask him a readable output. Um, yeah, I haven't tested it yet. I'm thinking that for medical considerations, it might uh overfit. so yeah, I don't want it to just focus well on

my few 1,00 cases when I'm going to scale it to 20,000 patients. I want to try to keep some uh uh human um reasoning behind Um, so once you have your prompt, you're ready to call your LLM. I'm using VLM I was mentioning. Uh, we start by initiating our endpoint. I'm using Podman. Um, it's a rootless Docker. Uh, it's good for shared Um, in this little snippet,

I'm using four GPUs. GPT 20 120B deploying it on port 9010. Uh the no enable prefix catching might save you some time. Um I won't uh elaborate too much on this but we can come back in in the question and uh yeah 0.15 docker image and we c the model to make sure we don't download it each time. So now our VLM is running on 9010 and

we can call it with any post query. I'm using R. This is the example of uh how I build my query. I have a function with my prompt coming in and this I build my body, retrieve the result and uh get it back from the function. So we now we waited a few time a few days few hours. Uh we got the LLM output back for all

our our inputs. we have our JSON objects and now I think it's the the interesting part of normalization the so one of the thing I saw was I was asking him if you have ulcers you put one in uh uh with the field name of ulcers and 20% of the time it will work 80% but 20% of the time will tell you presence of ulcers equal one.

So when you're trying to put that as a final CSV, it will break your thing and you so you have maybe five 10 fields you want to identify. So you have some randomness that you need to take into account. You need to have an automated system that will map them correctly to the box you were expecting. You could just recall the LLM, but it would be quite

a lot of calls and most likely you will need in any case one mapping at the end. So I was doing it with string distance regs but one of the thing is that at the beginning I was doing it like on 100 patients. I still had some labels I hadn't looked at. when I scale it to 1,00. Oh, there's some new cases that I didn't manage. Is

it still gonna work when I'm doing gonna do it on 20,000? So, I think that here um this morning the concept introduced was a human on the loop. It's not necessarily in the loop in the sense of we're not waiting for an for a human to say yes or no, but we do need an human to go over some of the logs. So one of the question

is how do you make logs that that you will be able to review easily. So this is my traceable mapping table. In the in the columns you have the five boxes I want and in the rows you have what was actually returned by the the LLM. So in here it's like on 100 cases. It's not that much. In the next slide we'll see a bit more. But

we already have some examples in the second part. narrowings example stenosis narrowing stenosis service disease service involved by disease. So these two should map together. And so I'm doing a little reggaps plus string distance matching. And I'm double checking that. Okay, these three first they correctly went into narrowings. These two they correctly went into surface disease. These two went into surface ulcers. So that's how I'm doing

the human on the loop the traceable mapping. And so when we go to 20K, this is what it looks like. Um, so you you can't be able to think about every case that's going to happen. You need to just try it out. But, uh, you I I was lucky uh, from my 1,00 to 20,000, I didn't need to change anything. It's more when I was doing the

100 to 1,00 that I needed to review some of my code. Um, so here in the first four we can see there was 2,000 that was actually the correct field from the beginning but uh almost 1,000 that were not. So if you're recalling that much it makes a lot of further calls. So, uh I think that uh you shouldn't just like recall 30% of your prompts. You

should rather find a good mapping. So, so the now you have your nice CSV. The that was the I think the the main uh conceptual part that I I found interesting. Um then we have bit more medical intelligence. Um maybe you were asking about ulcers. A second field is surface affected. The LLM told you yes there's ulcers but it didn't tell you about the surface. So I'm

adding some little rules. If you had one here you will need at least one here. A last part uncertainty and estimation. it is inherent in the in the domain. The the clinician might say yeah it's a bit in between. So you need to have a system that can manage this and in your evaluation it's good to take this into account to not uh underestimate too much your

performance. how I started doing this was three independent prompts low average high estimate. So let's say the the node says mild or moderate in the low one you would take milder mild and the higher one you would take moderate and the average you put 1.5. so in the in the first case I was doing independent calls. The nice thing is that you can see some randomness in

the LLM. Like you might ask him for the low and the high and the high will actually be lower than the low. Um however this does increase by three times the number of prompts you need to make. So in our scalable version we're just doing it in one prompt asking for three estimates. Um but on our like if we want to really focus on 100 patients we

can do three three calls independent. Then when once I have these estimates, if they all agree, we're good. If they don't agree, I'm flagging it. Uh it might go for manual review by the clinician. It's usually like 10%. So here we're a bit uh coming to the This uh should be clear by now. if you have more questions about it. So, build the prompts, call in parallel,

pause the response, normalize, rule-based medical The the cool applications of this treatment efficacy measurements on much larger populations. Uh so we performed it for around $300 on 50,000 nodes. Uh, this was with the OpenAI GPT5. Um, if you don't really have GPUs, I recommend it because it will be easier to get you started. However, if you want to move uh in a more intensive, it might be

worthwhile to invest in the GPUs. Um, maybe a box, you can count around $50,000 to get a box. so treatment efficacy and inflammatory bowel disease and for suicide prevention um increasing the reliability of the codified data. Um the the core experiments I did previously two or three days with four GPUs to do one year of nodes for 3,000 patients. We were estimating to reproduce it on the

Google compute would be likely around $2,000 and we're now working on scaling this to get to something that we can run on 1 million patients on their lifetime. Uh so here what I was saying on two three days I was doing every note full notes and now we're trying to do just extracts uh just the notes that are interesting the psychiatry notes uh and this is where

hybrid search comes into play. Uh so the the GPUs were kindly offered by the center for suicide research and uh collaboration between Mass General Bighgam and Harvard. Uh you have Jordan Smaller over here in the middle and my professor is over here in yellow. Some cool ideas I'm working on currently. So when I'm doing note selection for the the case of 1 million patient over lifetime currently

I'm doing some huristics to narrow down on the notes of interest. Uh we have too much notes to process them all even by just selecting extracts. I'm thinking that maybe agent could help me in there to focus on where what is the time period that I should find that I should focus on. >> Sorry. >> So, currently I'm doing simple aristics. Maybe the agents could help me

increase And the the really cool um more longshot project I'm thinking about is uh so for for prediction of mental health disorders, DSM5 has some questionnaires to evaluate personality traits. So personality traits can be measured in various ways. Facebook models you be by your likes. Um the the usual model you might hear about is ocean. You have like openness, extraversion, neuroticism. Each of these are usually linked

to one mental health disorder. So it's like if you're very neurotic, you might be schizophrenia down the road. Um autism has a bit of links with obsessive compulsive disorder. Bipolar has some links with impulsivity. Um so they are quite relevant in the medical domain as well. Uh DSM has a list of 200 questions and then they say okay if you answered in this question this goes to

this personality trait. You sum them up together. the the guideline is uh that the clinician should ask the patient to do it every six months over two years and if it's consistently high in one personality trait then we might want to investigate maybe the the patient doesn't have a formal disorder but some problematic personality traits. So this is how DSM suggests to use it and how I'm

thinking of using it is uh trying to answer these questions based on the notes. So let's say one of the question is I'm a risk taker. This gets mapped to impulsivity. in the notes you might have various things like the patient ran away from home. He's a kid. um he's in conflict with his boyfriend. Uh here the example I'm saying is jumped over a fire pit at

a party. You could think it's like not very relevant, but it can maybe help you answer the the question of I am a risk And so I'm thinking that this could be a cool way to model personality traits based on notes and try to predict better mental health disorders with that. Thanks a lot. >> Any questions? >> Yes. >> I have a I have a couple questions,

but I'll start with this one. Um, you mentioned chunking based on an arbitrary sounding three sentences or two sentences. Um, does chunking have to be done blind like that or um naively or can you use an LLM to sort of chunk for you based on uh thoughts, you know, or concepts? it makes me think of I think contextual retrieval where you're using a chunk but you're also

asking the LLM to summarize a bit the chunk above and the chunk below. however one thing maybe you need to keep in mind is that we're doing chunking for the retrieval augmented generation because the retrieval augmented generation like the rag the embedding part is more computationally efficient than calling the LLM. So I think the here we're trying to do this to not have to call the LLM

on everything. so it's not really the goal to call the LLM to help us on the chunking because it's computationally expensive. question here. >> Hi. Um my question is um with reference to neurology and uh depression uh connecting to suicidal tendencies. I attended a a a session on deep brain stimulation where they are able to detect uh depression and also to treat it. That means that they

are able to read brain signals and infer that there is a cause for uh severe depression. My question is can such uh information be combined with your AI efforts and conclude to a more concrete decisions and conclusions. >> Yeah. Um I would say these sort of signals are very uh real time and my signals are put more long-term. uh definitely can be used together. Um it does

uh make me think of uh maybe when you are able to measure uh depression, let's say you're already a bit in the zone of uh not acting a lot. Uh maybe staying in bed. Um and the important part at least for me is how to go how to intervene before you get into the zone. So I'm just uh um maybe that's one limitation of the uh brain

computer interface. uh you mentioned um more legacy uh LLMs and then now reasoning LLMs. Are reasoning LLM's runnable locally or when you if you want to use a reasoning LLM are you forced to use an API? No, the OSS GPT OSS are both uh reasoning models >> yeah questions. Anybody questions? Yes. Okay, there we go. For these um notes or the charts, how varied is the language?

like do you really need large language models or can you constrain what it needs to look through a small language model or something specific to just this area of uh exploration and then not have to worry about others. The reason I'm thinking is maybe you have a small language model for the IBD and another one for something else. Do they would they encompass enough information to give

you results? So should we should we use rather two different small language or one large? from what I see you do need a lot of general knowledge. So I'm a bit afraid that the smaller don't have that. Um in the IBD project it's probably not a big deal. you might be able um because you you mainly rely on a few synonyms like stenosis that means narrowing. That's

important that your language model knows about that. Other than a few synonyms like this, you don't need that much reasoning. In the suicide prevention, however, so I'm uh There's also like very various cases that you will come up into into the notes and I'm a bit afraid that if you fine-tune too much on one case, you might lose the generalizability. Um I I do agree that it

could be cool to measure the difference because you would probably get much more computationally efficient. So that's the the really key key thing that would make you want to go into that direction. So it could be cool to compare either a rag with GPT 12B or a small language model and you don't need a rag. So I think that would be relevant to test. >> Any more

questions? Questions. Well, thank you. Excellent questions. Excellent presentation. Thank you very >> Thanks a lot. Yes, it all worked. Use this microphone and test it. >> Hello. >> Have to hold it that close. >> Hello. Hello. Okay. Okay. >> Thank you. We use this one for >> Okay. Got it. Got it. >> And uh somebody should help you plug in, I guess. >> What's up? >> I'm

just I don't know. I was just >> Okay. Do you want to use the headset or is this working >> just so you don't have to hold it? >> No, they've held it. They've held it. >> They can, but I'm saying if you don't want to, you can use the headset. >> Oh, okay. Maybe they see you. >> Yeah. So, all you need to do is turn

it on and then that's it. So, there's no >> Yeah. And you can throw it in a pocket or whatever. >> It's whatever your preference is. But yeah, these are So, it goes like that. >> Oh, the headsets for us. Yeah. >> They've been just using micro If any go behind We go. >> You don't need this. >> Turn this way. Let's get that. >> There you

go. Okay. Try that. >> that sound good out there. >> Okay. Connect. >> You can turn that off until you're ready. have to have everybody listening and getting ready. Just the switch on top. HDMI. Are we There's online Okay, I'll use this for questions. Hello. Can you hear me? Okay. >> Okay. Yeah. Thank you. Thank you very much for attending to my um talk today. I'm happy

happy to have opportunity to present to the skill uh with this my third time attending this conference. I'm happy to have opportunity to present our work at a children's hospital of Los Angeles. So a little thing about myself, I work in the children's hospital of Los Angeles. I work in a genetic testing center, molecular diagnosis, tumor test, uh genetic disease test. So we have we we process

a lot of genomic clinical signing data also clinical data in our daily work. So that's why I'm I'm talking about my experience our experience with AI tools in these applications. So my talk the topic is a million to one and what to terms the generative AI tools in action for rare disease diagnosis and patient datomization. So I have to introduce two AI generative AI based frameworks and

the practice we use in in our work and addressing the two challenges in the precision medicine for the data science part. The first one million to one challenge means that we need to in genetic testing we need to filter millions of varants into a single positive variant for a patient and then generate a diagnosis report. The second part is the two terms challenge which about transforming the

unstructured clinical data into the ontologies and dictionaries. So the first part is about the million to one in the genetic setting data processing. We are trying to filter the millions of variance from patient and So precision medicine which was really get a heart in 2015. It was first well known as the molecular At that time the president President Obama proposed the precision medicine give promise that uh

everyone patient can get a diagnosis and which will direct their treatment and get the treatment affordable and precise. So usually the genetic molecule diagnosis runs start as the clinical lab where they do the leis sequencing data. This kind of lab in the US will be cy certified and also ke certificate at least to be able to offer this test to patients. then followed by the data processing

including the varant calling and interpretation for the pathogenicity and finally science molecular diagnosis Now the beauty reality is that we know we can get a $100 genome sequencing. The technology is very much better and faster and a very cheap chip uh in the clinical sequencing. The clinical sequencing have two main two type of sequencing. First is the next generation sequencing which was short read mainly provided by

illuminina sequencer complex genomics team and now the rush they have a new sequencer. So basically the most of the companies they are now claiming that they can offering the whole genome sequencing as a $100 or a little over uh per patient per human genome. For some patients, if the next generation sequencing data was not positive, they may need to go to the third generation sequencing using the

long read technology from nanop pack which will detect some mutate disease causing mutations missed by the neg sequencing technology. And uh so this graph left side shows the cost what the cost was coming down from over 10 million dollars in the 2020 2021 earlier the start of this century and after the 2007 we get the necessary sequencer from illuminina the per genome cost are going down rapidly

and now this last year people already are offering $100 whole genome sequencing and with the new sequencer to the right side the high throughput in the illuminate you can generate 8 thou,000 gigabyte data a day. So basically yeah this is a general flow for the clinical sequencing in the molecular analysis uh service. First the celia certified lab will do the gen sequencing to generate the whole genome

sequence or who sequencing if the whole genome sequencing WGS you will get a 50 gigabyte gigabes of data. Then the secondary analysis will call the variant which are changed in the patient against the reference genome and this will retain you of four to five million variance. If you are running whole genome sequencing uh after whole genome sequencing we have a tertiary analysis that where you trying to

filter down from the millions of varants and identify one positive variant for the disease for the phenotinical phenotype in the patient that's yeah why I call it the millions to one actually in reality the diagnosis rate is only 25% at most 40 50% that means you in the lab they will sequence call over 10 million variance and finally get a one positive varian call to the patient.

So it's actually 10 minutes to at least. So the challenges in the precision medicine for the diagnosis is not come to the ugly side that we we can generate data very quickly rapidly but we lack a way to rapidly generate insights to cause penic varian analysis. So the typical T run time is still two to six weeks in most labs including our center. Yeah. Photo analysis the

secondary analysis take only hours for patient but the touch analysis interpreting and reporting part could take weeks you in most lab. So basically now if we assume we are doing the whole random sequencing at a 50 coverage deps then the the no s plus for illuminina can generate the data for 150 genomes a day but secondary analysis is still slower. If you are running the GP even

with four GPUs four GPUs uh you still need 13 millions at over two hours to call. If you go to the high end Handroid that used take almost one hour. So the secondary analysis is much slower. But even slower is a touch reanalysis that usually take at least one to two weeks to run the variant annotation interpretation and review by the clinical director to get to decide

if use if this case the first positive mutation is a variant. So the the leg is really that the sequence data uh interpreting part. So uh we yeah develop the AR for metal or AR for rare disease interpret which is the AR based framework for the tertiary analysis in clinical uh this framework can work for the two genome sequencing data who sequencing data gene and gene panels

and we are using AI trying to emulate the clinical geneticist the reasoning pro u process invariant pro uh pization and framework is hard is mostly automated so it can rapidly prioritize the disease causing variant from the four five uh variance called per case per patient framework I work on metro disease so it's tuned for metrono disease but it's readily available applicable to other rare disease as long

as they are genetic diseases is so the core engine we have the LM model which I started with Google's German 2.5 uh in last August and later I gradually ported them to the open source queen version three uh local model the thing yeah it's supported by large log base including those knowledge base from the large language model from our diagnosis report in the past and also from

Mic another disease database we have been running for 10 over 10 years and finally it will generate the structured clinical report that it will prioritize ACMG classification for the var pagenic varian which are causing the disease based on the ACMG's and the guidelines and also have other diagnosis information about how the genotype are linked to the phenotype and to the disease diagnosis and also give details of

the rationale in during this reasoning to come up this conclusion. And also the report will have some disease diagnosis, differential diagnosis and some treatment and disease management recommendations. Yeah, this graph diagram shows the architecture of this citiz basically you have the input from sequencing you have input from the year that the the patients clinical health record and also you have knowledge base from clean at GMD and

other annotations. Then we run the Binformatic pipeline as the euro to annotate each of the variants in the and filter down the variant throughout those and luckily to be functionally important. Then we go to uh feed them to the AR generative AR code which include the PR engineering layer that will take the data from the input take the knowledge base which are relevant also take the API

call to the large language models server run the German line and other large language model to generate the diagnosis. report. So basically yeah this architecture and this one shows the workflow which are basically the same the same information in the architecture and it's only that if you have negative case you can you may go back to reanotate your data and try to run this AI powered analysis

framework again. uh this AI based framework is was derived from our previous work. We we call the PCV machine learning workflow where we are trying to do a phenotype centric variant classification using machine learning algorithm using similar data. Yeah, you see the middle about the data from the patient from NGS lab offer from the uh clinical record in step one they are they will win through the

and variant filter down in the step two then we do deep annotation variant annotation with all kind of well-known engines like VP P and like a LGMD clean and population LA frequencies and uh more from PL you can plug into the knowledge base and uh then at step four we get the in uh short list variant with deep annotation from all the sources. from here. In the

past, you will go to step five to run the machine learning model uh to to find the to uh candidate for the pagenic variant and then send to the clinical director to review and decide if they found this call is a var is the positive variant. But as step four now we use the input file also generate a gen AI prompt for the framework. This summation we

implemented starting last August. So this one is about using the same data as the machine learning module now and uh you run the promp lang models to generate a different type of clinical diagnosis So this system was tuned for rare disease you rare mitochondrial disease. This is why I'm trying to show quite a complete case report for the A8124 which this report was generated for serving the

clinical clinic clinician is to help them in interpreting with by providing treatable evidence and reasoning process. So first you have patient summary with the clinical data. Then the second with the HPO ontology encoding which are essential uh to to link to the disease profiling and the genotypes and part two is the primary genetic finding about which genes varants was found to be most u relevant for the

this patients importance and that give a var detail including the gene names the exact changes the effect on proteins of the for this gene and also is because this is a mitochondrial it have something special called home plasmmy which only found in mitochondrial DNA mutations and then we have the clean water annotation and hg which are the most reference against the known positive variant the database for

this variant yeah see we have report from clean where they was reported pedagogenic but hgmd is hgm doesn't cover metro variance so it's negative for hgmd database then the part three is call actually the most important part for the generation report is very the pathogenicity. Yeah. The first is it said okay classification is pathogenic and the rest evidence is that clean causes the pedogic by other labs

multiple labs already cause pathogenic although at GMD and J&B are not not so positive and the second part get a detailed evidence code for H ACMG evidence code based on the ACMG these uh guidelines which are the official guideline for the molecular diagnosis data interpretation for this variant in metron DNA. You have the evidence called PS3, PS4 which are strong evidence based on the data you provided

also based on the data. The large line model are pulling from its own knowledge base putting from the vector database we provided back context and also some website web search at real time and uh this based on different type of changes effect of this variant PS3 because we have new functional studies in model system to provide that these variants are causing functional change which are disruptive for

the functioning of the protein. The PS4 is that it is already reported in multiple time patients. So that is get same disease same same phenotype then you have a moderate support evidence based on it's not same in healthy population you got P2 if you have a different another variant slightly different same position with different change of T2G rather than T2C but at the same position so this

give you a PM5 supporting and then other modes breaker seen as the PSP PV4 is breaker because it's known that this gene was associated with this led with the phenotype scientic and uh this shows the older version before we put in the mito variant specific rules in the variant scoring you see that you get a few uh I mean is the code p you don't have a

PS4 and PP4 we have seen the last slide also called something PM1 is not so specific as PM5 because PM1 means that this area this domain is important pain but the PS PM5 say okay you have a exact position a slightly different change at the same position so it's more specific then you don't have the ACMG guideline tuned for metondro disease so that I mean that yeah

the better you define the rules to the large langu model the best it can utilize the data and reasoning to give you the evidence code to support the classification. Here I give the disease name diagnosis also the other less likely secondary diagnosis and put the liter relevant put the abstract from the literature based on the data. This supporting evidence and his last section will give you some

therapeutic suggestions meas disease management suggestion which are as for information but it's not clinical advice for the disclaimer. So basically yeah we have the detailed introduction about this uh this framework here we val we benchmark the performance first we benchmark by creating a simulated cohort with CT8 cases we use the published cases from G4 GH we spark the positive positive variant into a negative VF file which

are the what we generated from the sequencing and the filter down uh in the secondary analysis and then we run this framework the AR framework is get 40 64 them correct this translate 94% sensitivity and for diagnosis or disease names we get 49 correct which are 72 2% accurate. Actually, this is a high accuracy because in our lab in the past, we usually don't give the disease

name because it's hard to get a right this name based on the patient's clinical phenotype variant. So basically this means that we have high sensitivity and we also get the disease name quite correct and the each case take less than five millions to complete actually. So it's quite fast. We also compare with machine learning based the PCV classifier that that classifier get 84% sensitivity and about half

of them are posit are true positives. So it's doing better than machine learning alone which which are not powered by the reasoning from large language model. And we also compare with some commercial solutions, third party solutions which we checked the benchmarking last year they released. So they those commercial solution last year at least they are retaining your list of candidates rather than a single um pogenic uh

variant call and they usually give 90% in the top two pigs and or 94% in the AR Mario but 74% only D AI which are if you limit to the of wine. They didn't give specificity but because they usually pick at least two of them at most one of them are correct you have less than 50% true uh positive here the table we we summarize last year

based on the commercial solutions of different companies the first one which uh bought by the JDS comp for $50 that they gave some 90% in top two sensitivity. The second one is more half of the academic one. So they get 66% in top one and 94% top two. So the at least for the top one P is 66% is less than what we are getting 94%. and

dark AI is from another company that gave us something uh slower uh is 74% to So basically yeah our machine learning uh PC here do something similar to them but our AI based framework AR for metal we get a much higher sensitivity that we tested. So basically yeah in summary this AR based framework is effective for prioritizing the cy varants for genetic testing the sensitivity is high

with 94% at least for the simulated cases and this provide a traceable very structured and comprehensive reporting which are good for the clinical directors to check the evidence to decide If this call is correct and the report is human readable then the machine learning results it give very detailed all kind of evidence with rationale and with evidence in the report and we are running it importing it

to the open source Lang model and is is showing that the performance also not so good as the German line but it's comparable by ting the prompt. So my next section is about data transformation in clinical data We are trying to use the AI to transform the unstructured clinical data free test note into standardized to enable the cross comparison. The ultimate goal is we want to get

value from a structure data either from internal record in hospital or from the publications. uh for the director uh objective I was uh uh asked by journal to write a paper. That's why I was trying to initiate machine meta analysis to quickly go through to create a virtual registry and use the AI based uh acceleration and we try to set up a framework for transforming the clinical

note into standard dictionaries reliably and with high accuracy. for that special topic I was trying to study the le syndrome which is a very rare disease you see in V 34,000 buses only so the it's rare and usually the each publication each people give you very small cohort of few to thous to thousands cases only you need to merge many publication to generate a large cohort so

that you can have a Well powered large cohort to do meta analysis and also the data every people also reporting data differently different uh so you need a lot of effort if you manually curate the data to unify the for example yeah for the for for the collection 1,600 cases the age came from 21 different terms of different table column is by the alo the different columns.

If we go to the phenotype coding that is more heterogeneous for one single phenotype that is decrease the activity metro complex one that also use 24 different terms that we need to to recognize and not only the headers 29 terms we also need to convert the original coding of the values to a standard term to this decrease activity. of complex one. Uh so we use so we

deise that AI powered harmonization pipeline which uh this hybrid pipeline combines the generative AI reasoning with human loop review to ensure the quality but this is rapid and we can gen we can process the over 2,300 published cases from 46 papers is only under two weeks by myself in two weeks and the accuracy is high. We get 90 over 90% phenotype uh accuracy after manual checking and

some refinement of the AR generated mapping. So we so basically in two ways I generate a cohort with harm data with 1,600 cases and they are deeply annotated. Each one I get over six phenotype cing these cases to get a rich charact of the phenotype. So this diagram shows the rough workflow. From here we do the this research as a power made with kind of word and

then manually pull the tables the kids level tables from the papers and load to our database. Our website our website science we get the data in the SQL database which are heterogene which are inconsistent between different papers then in the step two we use a based framework to do transformation first we start with gemini 2.5 in August which was the best at the time then we we

divide two pro for the data The first one focus on age related information. The the pro AI method for age and the second one is more comprehensive data transformation for all kind of phenotypes hydrogenicity model inheritance also the genotypes information. This these are the two pros that we is device and then we generate the harmoniz report from the AI transformation. We load to the SQL database to

our website again which are much more structured and standardized. Then we step three we use a phenotype humid loop quality checking refinement using some tools we use in the past for the phenotype mapping for free test to the human phenotype ontology mapping in the past which was sematic similarity based but this time we use it to do the QC and refine it and we finally we get

the final water cohort in the database with over 1,600 cases is with 11,000 entries that average 6.5 phenotype for each case which is a good phenotype for for the cases. Then we do meta analysis, survival analysis, stratification analysis by intersecting with the genes with the genes with inheritance with the sex information. Yeah. Yeah, here I just show that we originally we select German 2.5 because we compared

with C GPT 40 with we compared with open source L models but they cannot handleing the phenotype data we saw in the publication. So we pick the geminite to start with. This is one prompt for the age processing. Basically you say okay you the rules we define the rules to be simulating the clinical geneticist trying to transform data. We give some general guidelines to request that to

be uh strict on the data fidelity avoid any speculations. Basically we want to avoid hallucination as much as possible. Then the input data format we specify we generate from the SQL the columns we defined and we let the AI know the data columns and then the output format we specified to the AI. After that we have the execution which have list each step that we want the

the AR to do based on the data we provided based on the rule we provided including HP term mapping including the medical disease diagnosis criteria criteria based on phenotype based on the the different type of phenotype how important are these required or sage option you or it's important to diagnose to make a diagnosis. So the stage one yeah we process the raw keys collect from the 46

paper 2300 cases this will process the processing take one week at the German because I write the batch by batch at the the studio studio which is manually then we do the man human in the loop review which are manual and also including sematic similarity search which take another week. So after one week I get process them to get a 14,000 HPO based entries and also I

threw out some cases that was not very rated in the original paper then that 1,600 1,679 cases in the final virtual cohort. During the process, the AR get the AR first run get 77% of the term accurate in the HPO term name and HP ids. Then we run through the human refine in the loop refinement improve the percent to 90%. The AI also inferred some data which

are not reported by the author but the the paper the context allows the infer the information by the AI based on the knowledge base. So for example psyosity we have almost five foot more called in the AI generated data. So this is the raw data which are very free text. The right side the free text the raw data the left side AR transformer data. So basically clinical

data we get some code called by to the phenotype names. We also get some biochemical lab testing like which are numerical values. We need the AI to convert them to phenotypes based on the rule based on foundation models knowledge about how the lab testing should be translated to is this normal or is this decreased or is this increased this kind of information but AI can do this

quite well during the also we are in the original paper has been the lot of abbreviations in the publication is that you see the middle there are dozens of abbreviation case you need to expand them to the four fre names if you are doing manually that each cell many cells you need to do it manually to use this key pair but with AI it is not this

mapping quite accurately. So finally yeah we have each case the AI will process data based on the abbrevation key you provided based on the HP mapping rules you provide in the program and other rules based on its own knowledge based on the internet search that translate to the middle the AI standardize the phenotype code from the raw phenotype in the right side. uh here shows they called

the AI was doing the phenotype mapping from the raw clinical node. Most of the majority of them they can derive from the clinical descriptions although they are transforming to the standard names used in the dictionary but there are some the second is some biochemical lab testings. It will translate them based on the knowledge general knowledge. The numerical value will convert to abnormality cause decrease activity, increase activity

or something like and also there are something other not a very direct clinical information also can be transformed by the large languidation quality checking. We see that group A group A point 2 this machine where the AI get the mapping correct very accurately in the first run that assumption over 90%. uh and in group B there assumption of 10% of they get the the the phenotype terms

right which because the terms are long they have better measure to the clinical node in your raw uh clinical node but HD actually meaning this only a single ID with six or eight digits so this had a hard time to match them correctly and so also In group C we have something uh the terms matched are quite far from the standard terms although sematically they match to

the HPO ontology dictionary that way we use the sematic similarity search to do the refinement by human loop in group C then finally we we we improve the accuracy to over 90% from the AI based mapping So basically after that we get this and we load to our website which we pro we have a web resource for the virtual cohort let us use to access and search

the the cases with different clinical phenotype or combination of phenotypes and check the details. So this is the tools we have and then yeah after we transform the data to standard terms now we can do some uh clinical gen charact work cohort to do meta analysis which is four times larger than the any of previous reported uh cohort for the le syndrome. We have no much high

percentage demographic information available the the sex the age we the age of onset we know that we not know 41% are infantile or earlier 34% are childhood so that this disease quite early onset for the lyn room and inheritance we have the most of them are autosome recessive and some of inherent parented for the mother of the mitochondria inherited 78 5 752 of them and after we

standardize the phenotype and we and we use the gontology tree to consolidate the to some more general form we have we can identify the hermark phenotype which are important for diomic diagnosis which later yeah has make diagnosis on genomic changes based on the phenotype in the cases and yeah for yeah most of all yeah interesting is that we can now can do some survival analysis which was

not was not possible with a small cohort in the single paper now we have we merged we have 79 cases we can calculate the exact survival time in in numeric numerical value of years based on the age of onset age of death. This give us which the power. So with this we can okay the we can see that mitochondrial translation changes are associated with the the worst

the pronosis because the 84% of them will die and the for this group if the they will the disease started under three months which is a point two three years only very early on in in the love hood and they survive very very short time And we also found that the TRMU gene if they have the change at the second position of CDNA that this will have

survival time only and four months which are very poor and then yeah we get we get this conclusion because we combine this from multiple papers that we can reliable to give this conclusion and for the con complex complexes one through five. We see the the complex five have the worst survival only less than two years. If this affecting the complex one or complex four and the survival

time much longer but still under four years. So this le syndrome is quite something unfortunate the short survival time if they are affected and we do the survival analysis based on the larger merged the cohort with 700 cases have exact survival years. So we cannot stratify by the the genes by the age of onset by the sex and by the gene function group. So for the sex

we see that the female and males they have a similar onset to the left side but after 10 years the female patients will have better survival rate than the males after 10 years this is we have this conclusion after we get a larger cohort and this one about different gen is tru to the to the bottom you very early on had a very very low survival after

10 years. But if the gene was a the mutation was affecting another or ATP6 then survival time are much better even after 10 years this they have over 60% survival. Yeah, from this work, yeah, we have publications from this work. The first one are the Lelay syndrome cohort that we published early last last month which we process 2500 cases actually with 126,000 data points we transformed with

AR with German line and we published this and later on we expand this AR transformation and now we ported is the prime of the AI framework to open source with with the queen version three with a local 30 billion parameter model that we laptop that we to transform the data we also get a very good transformation with age information phenotype information the inheritance modes and also the

genes and mutations using the open source large language model now. So that we have systems that have not only uh powerful also low cost to use. Yeah. This in comparison. Yeah. Two months ago there another paper by Germany lab that they do something similar for building the water cohort for mitochondrial disease. that paper they they published the first paper four years ago. Then in the four years

they added another 3,900 cases to the virtual cohort in four years and you have they have over 10 people work on that. So because they use a mly work uh so you see that it's quite time consuming to do the transformation for virtual cohort in metano disease if you are not doing it with AR or something computing tool to use yeah actually yeah we do the transformation

11,000 this machine I started in mid December using the open source based lang model basically I did all the transformation and hum uh validation by by all by myself you and one month also yeah is over 10,000 cases with AI but from the data yeah I can reliably yes see I can write a paper the second one we already completed the writing based on the data so

the AI is very powerful very fast to do the data Yeah. Also, yeah. So, basically for the data transformation, yeah, we have the what to terms framework in generative AI get a quite satisfactory performance that I can trust based on my experience and my knowledge in in curating the clinical data. And uh for the lacing room we already generate the largest virtual cohort were almost 2,000 cases.

This four times the largest than anything in the past and the disease cohort is also larger than the Germany group generated and published as a decent journal. And this framework can work on any rare disease from literature to do the data mining. So we have this. Yeah. Yeah. So the prom was attached in the uh supplemental of the first publication also going that after I complete that

the two two papers writing uh going forward to refine to reduce the manual effort by reduce effort in the human loop part further to to improve on the pipeline. This is why you see that we have a free text clinical node summation non-standard summation even spelling errors in the raw clinical node. Then we the AR for HPO annotator to do the Then if in the first run

it will run a phenotype based on the understanding based on your attribute uh definitions you provided and al also from your pro also from the back end vector database then it will get a further path. uh the this will transform the draft retest to the draft phenotype snippets and also infer to the standard HPO ontology term and RDS the terms by review is quite correct but the

RDS are mostly wrong so that we provide as we seen in the past pipeline so what I did is I get the human loop uh part the sematic big similarity search I used in manually in the past. I run it in the back end prepare input for each term. I picked the 20 best match based on similarity sematic simulity search and then I ran the second time

to review the up to 20 candidate for each ra phenotype to 20 up to 20 top candidate phenotype HP term. Now I have theatic search get the HP is very exact correct. I only need the AI to pick the right turn with the right LP. So now we have the best P best P by the AI which almost all the phenotype RDS are correct now although they

are meaningless they are only RDS but now they are correct. Of course the phenotypes in LPO terms are not very correct. So I have system now this machine you see the first step machine 36 second with a local open source large model running a t on my laptop and the second second run some machine still and 41 seconds. This is the thing is take less than two

two millions to do a batch of input up to 10 or even 20 clinical terms and map them quite correctly without you doing anything. You just wait two minutes to see the results and you have table generated. You can see the table to hear the way something to to do it. So basically that's it. So together I think now the Lang model and open source tools open

source spirit we have open source LA model we have open source tools by information tools other tools in the we can combine them to democratize the data science part for precision medicine data processing we no longer said okay we need to rely on the commercial companies solution provider They run those hardware state they lot with millions dollars to set up lot of team to to maintain it.

But if we only get want to get some quick answer, we may do it so locally without data going out without privacy wiring. But you can run something hardware at yourself for myself this together and $5,000. The most expensive is Lenovo laptop with 24 CPU with a 55080 GPU. These are moderate with only $3,000 for my to set up those things to run this web server website

at my home and you I won't need to get data out to to them to get the agreement of data transfer to the connection. Yeah, suppose yeah we can get a similar infrastructure to the to the clinical production to to the hospital to run get something set up similarly without very expensive hardware but you can some performance. Of course you can always get a better performance larger

L mode large larger model or higher in hardware but it's not absolutely essential if you can get the rules get the data input correct I think that's the thing about my talk this timeline in August that started to do this work because the journal asked me to write something without paying application fee to them so that I r to generate this power line to do the data

preparation and generate the power line I first and first start with the with the German line with a paid solution then I try to port to open source solution first to the ki which did well in December but I didn't do much because it's still paid service and later when I got the new computer with good GPU start porting to local model on my computer. So in

December I set up the data transformation part and in January I get the genetic testing data interpretation part work you see the the landline model is quite more quite a small langu model actually that's yeah finally yeah all kind of people in the children hospital and in our team and also the collaborate the children hospital that the children all the bit of the de also all the

people James yeah yeah we worked together for many years also my wife is listening to me here so also yeah you talk that's my talk thank you for your attention >> it's turned There you go. Thank you so much. Sure. >> So, Leewang, uh even though right now you have a a trained uh um uh uh retrieved augmented generation with Quen 3, uh have you thought about

creating your own micro LLM for doing just this work? have it on my computer. I have it have web interface on my computer have web interface call to do the transformation at the website the regular website the for the uh for the molecule diagnosis report I need data preparation as I show in some slide PCV part that we need to run offline but we have seen in

the past our hot that I set up to run. Yeah, but is running as a calling to the large language model is it had a API interface if you open it with a key. Uh I'm not work yet because for now I put router very long pro over 3,000 word to put the things in all things in my mind. I I think they worked well. So, I'm

not trying to go to the Atlantic programming >> Sure. >> Well, maybe that would be something if you if think about if you want uh like a an idea for a research grant that to to take that on. But yes, but this is uh great work. Yeah, I try to Yeah, I'm trying to see. Because we have been doing processing all these data which are very tedious

even no matter supporting the clinical director in our hospital or no matter the control database that are trying to make to the public open I said to the world you take a lot of time to something passion to do those kind of work it drew you crazy to make the data manually but now you have something that can do the work for you. Hey, um I saw

on one of the last slides a chiwen 30 billion parameter model. >> Uh I saw that um when you were doing porting on open-source models. >> Yeah, >> I think you mentioned that you were using chiwen 3 with 30 billion parameters. >> Yeah, that's the model I write. Actually only the 30 billion model as quantized toward the middle Q4 is not high density but it works well.

This surprisingly works well actually I run the new 3.5 is much slower but it's not much more better because I think because I run it many times have the history have something yeah in to the contest in the in this instance running on on computer. Did you test um other models? Uh was there other models that were not working for you or actually I run the the

open source from open AI they released some machine that that's not doing well. I also run the granite new model is not doing well. The main is that they are not generating report in the format as specified as I show here. I want in the consistent format as I got from German as I got from queen but those model they can do something good maybe but they

are not generating report in the format I I wish I I see. >> Oh thank you very much for the presentation everybody. >> Thank you. Thank you for >> You seem to have you want to do it. That's fine. All right. We'll turn this terms. I teach >> I first introduced a lot of natural language processing to one of those started using natural language processing and then

of course the next cutting edge part of that was language. >> Oh yeah yeah How do you turn this on and off? >> He just brought >> What's wrong with it? >> No, nothing's wrong with it. I just don't know how to turn it on and off. >> I'm assuming you want the >> Do you want microphones or this ear thing? >> This? >> I like microphones

better. >> Good. Excellent. Then we don't have to worry. And you're the last presentation, so we don't have to worry. >> Here you go. >> Hello. Hello. Hello. >> David, go out to the hall and bring people into our talk. >> Hey, y'all. Y'all satellites are here. >> Open source and space. Oh, >> I want to hide this down here. Will you promise me not to kick

this? these are stickers. You can hand them out to people who ask questions. >> Excellent. I can do that. They get >> very nice. >> Right away. >> I'm going to take this and put it away. Put this >> for me to give away. >> It went away. Joys medicine. Yeah, we got to pack the house here. Although the last time I did a talk, I think

it was in this room. I had like 20 people maybe. So, it was kind of a disappointing audience in inside, but they asked questions for two hours after the talk. >> The AV team that I work on shut down the feeds and walked away like really. >> they do that same thing. They said we're you're done. Pull the wires. >> There we go. >> How long is

the talk? >> It's an hour. >> But we're the last of the day, so the only people there to stop me are Ron here and Dr. Smith upstairs. >> Testing one, two, three. Testing one, >> Hello. I am holding my microphone like I'm singing. >> Okay, >> there you go. >> Just close enough. >> Oh, this is this is a strain on my arms. >> Is it

ready? No. >> Uh, then I have little cards like 10 minutes, five minutes, one minute to warn you. Oh, she's not listening. That's okay. I have little car cards 10 minutes to go, five minutes to go if it goes that long, but it probably won't. And then at the end, you can say questions and I take this around and wait till they have a microphone because we're

recording the wonderful questions. Okay. Done. All >> Test. Mic test. Mic test. That sound okay? >> You sound good. I sound good. >> I sound excellent. >> Yeah, it's a co co-presentation. Yes. >> Yeah. I'm just I'm taking >> Yeah. And got the the headlining spot. >> We don't have to stay for the whole day. because I I parked outside like two hours. So, I need to

move the car. >> Oh, you should go move the car. You've already seen this. So, >> I have not seen >> You've not seen this? >> You didn't come to Tokyo? >> Sam, this is the one hour version of >> this. This is the extended version with all the nitty-gritty details and cool anecdotes and all sorts of wonderful things. And >> we also explain what a satellite

is, which I'm sure you know. right. Hello everybody. We're going to go ahead and get started with the talk here just after 5 at the Southern California Linux Expo. Um my name is Michael Starch. This is Ines Quidder and we will be talking to you about five satellites in five months. how we delivered rapid, reliable, and open software. So, first before we jump in, I want to

state there's plenty of time for questions in this talk. So, if you have questions, we want to hear them. We have some incentives for people to ask questions that you may or may not see. So, I don't know, maybe ask some questions. I'd also like to do a shout out to all the volunteers and uh staff here at the Southern California Linux Convention. you know that um

or maybe you don't know this uh event is mostly volunteer run. So shout out to all those volunteers working hard. Uh we've been volunteering for the AV team which is amazing and I just did the thing that I was not supposed to do. So um anyway, we'll get to that in a second. So what is our agenda today? We're first going to talk to you a little

bit about the introduction to the PRVs um software. We're going to talk about what are cubets, who are uh uh who are the proves five and what is cubat software. Then we're going to talk about the development strategies we used to develop this software. So we're going to talk a little bit about what MVPs mean in this environment. We're going to talk about how we leverage the

tools available to do development. We're going to talk about testing because everybody's favorite topic is testing. And then we're going to talk about how we streamline decisions. We'll close the talk today with a bunch of lessons learned where we'll talk about the plan for changes uh practical concessions we had to make uh everybody's favorite beware of state and how we learned to trust the team. So I

wanted to talk a little bit about who we are before we uh jump off this page. Uh again I am Michael Starch and >> I mean s >> we are from the opensource space foundation. So we're talking a lot about the proves project today, but that project, the hardware plans and the software that we've developed is released to the public as part of the open-source space foundation.

So everything we talk about here today, you can go find online and have fun with and possibly even launch to space. So let's go into the introduction. The first question is what is a cubat? This is a cube set >> and a cube set that I probably just damaged by kicking it down to the floor. >> Yeah, hopefully. Free vibe testing. There we go. We did it.

All right. So, for those of you who aren't aware what a cube set is, it's a spacecraft, which is really cool. So, this talk is about spacecraft running open-source software. These cubats have everything that you expect in a normal spacecraft. They have attitude determination. They have communications. They have control software. They have payloads. And they have smiley faces on the back. So these cubats are really full

functioning spacecraft. The thing that makes a cube set novel is its size. So you can see I can hold this cube set in one hand and this is actually twice the size of the smallest cube set that most people talk about. This is what's known as a 2U. And you can see on our picture here we have some one use and a nice little cubet plushy to

round out the mix. Cube sets are small. They make it easy for institutions like universities and other groups to launch small high-risk payloads into space so that we can study and test things in space without the you know traditional multi-million dollar cost of a satellite. So who are the proves five? The proves five are five universities that conspired if you will to build this particular cubat. So

we have the slugs team from University of California and Santa Cruz. We have Calpali Premmona Bronco Space. We have the Texas uh state space laboratory. Um we have the Colombia Space Flight Initiative and we have Northeastern. So these are the universities that work together to build this open software and open hardware to build this kit that we can put into space. So our project is about how

we built software orchestrating five separate uh universities in three separate time zones and did it all fairly rapidly. So to take another look at the team, this is our team. Um our team is composed of a number of uh people that you can see here. These are pictures from integration and delivery of the satellites. And our team spans all of these universities. But we came together to

build this spacecraft. So I'm going to hand it off to Eness to talk a little bit about the project in more of the numbers and then we will get into spacecraft software. Yeah. So uh the proofs project started a little less than two years ago when we got a grant from CSLI the cubet launch initiative to launch five satellites. Um since then we spent about $35,000 uh

USD which is very very cheap for five uh flight unit satellites. We soldered 540 solar cells, 95 flight controller boards and a lot of iterations. So we started with the V5B and then V5C and then V5D and then V5D prod and then V5D prod one and now we're on V5D prod 2. uh right here. Um you it's fun it's fun to see all of our circuit boards

because that they're like green green and then we had like an era where we had black ones but then we were getting so close to the deadline and the green ones come like a week faster and we really needed that so they went back to being green. Uh we also uh crimped space grade wires. So we crimped about 700 uh wires. I can't hear the sound of

the crimp tool anymore without like looking around me for the evil crimp man. Um, we made nine engineering models and five flight models across all five universities which you can see there. Those are the flight models. Going to go quickly over kind of the missions of each of these satellites. So over here on the far uh what's this direction on the far right uh we have the

UC Santa Cruz satellite. Shout out to UC Santa Cruz um UC Santa Cruz's first satellite. Their payload is um a ballast payload to look at the orbital lifetime of the satellite in orbit. Then we have Columbia University. Their payload is like a little deployable door and a little lion will come out. So it's like a selfie stick. It takes a selfie of a little lion in space.

Over in the middle we have Calpolly Pomona. Shout out to CalPoly Pomona over there. Um they have the TU and so because they have two they decided to go with two payloads. Um, there is a radiation sensor payload as well as an S-band radio payload, which is actually very fun because the Sband radio payload was developed by SATNOX, which is another open source project that we're helping

test. And that's in collaboration with the University of Hawaii. And so, as we're doing payloads, we're also contributing with other universities and other open source projects, which is cool. Then we've got the Northeastern satellite, they also have a camera. And then the Texas they've also got an open source payload called Mosaic, uh, which I would highly recommend looking into later. They do like for university teams, for

students being able to swap out sensors in kind of a space grade way. And so they're also doing a radiation sensor payload, but as part of an open source platform. And so it's really, really cool that we're all working on these together. But it's also like we're doing five satellites developing one software stack and all of these different payloads, and we got to do it all together.

uh talking a little bit more in general about what a cubat software looks like. Um here's a picture of a ground station that one of our team members from UC Santa Cruz uh is working on um and picture of our satellite which uh not to scale um how it will be when it's orbiting. Uh what the satellite has to do while it's orbiting the earth is it

gets signals from a ground station. It gets commands and it also downloads telemetry and events and all of this data from the satellite. some other software that uh the satellite has to do is has to detumble a spinning satellite after deployment. And so often uh the ISS doesn't do this anymore, but they used to like throw the satellites out like a football. Now they have like a

deployment mechanism and like a robot arm. Uh but and then if you're on a rocket, they'll be deployment mechanism, too. Often the satellites will be spinning after that. And so you want to uh orient yourself based on like the Earth's magnetic field using what we call a magnetometer. Um there's also light sensors to know where the sun is. Some satellites have cameras called star trackers which track

where they are based on the sun. Um, we didn't have that. Uh, it was too much development time, but we also didn't need it for the type of dumbling we were doing. And the reason you want to detumble a satellite is to communicate with a ground station. If you're spinning really really fast, it's really hard to get a radio link to the ground station. Uh, that is

really good. And also, if you're taking pictures, like many of our satellites had a camera, uh, not a really good quality picture if you're spinning really really fast in space. Uh so we have a detumble mechanism. We also manage operational modes. Um crazy thing about satellites is uh you can't turn them on and off again when they stop working in space. And so you have to make

sure that you think of all the operational modes, all the things that can go wrong and that it can uh solve any issues that come up for itself. And part of that is having a safe mode where something non-nominal happens or if the power goes below a certain level, we go into like a power saving mode. Um actually Sam over here in the second row he was

a big part of our mode management system. Uh we also have integration with ground station commanding. So all of that bits about event and telemetry that I was talking about. And then enforce ground user authentification. This is especially important with our satellite. Um if anyone here is a radio amateur part of our payload is actually that any radio amateur if you've got like a an amateur license

you can ping our satellites and you can ask our satellites for a joke and they will downlink the funniest joke you've ever heard. I promise. So, if you don't have a regime amateur license, maybe consider getting one. Uh, we're launching in April, deploying in May, and then you can get some jokes. Uh, but because we're telling people to ping our satellites, we don't want people to ping

our satellites and start telling our satellites to do things that may be bad. And so, for that reason, we have like security and authentication on our satellites. Um, and if there's any questions later about the stuff on our software, we're more than happy to talk more detail. Um but we're more talking about the development of the satellites. Um this is just a a thing. Okay. So now

uh we're gonna talk about our software development and why did we develop software in five months for five satellites. So the proofs kit is um a longunning open source project. How poly pomona has been developing it for a long time and they've launched many satellites with it. But as we were developing this project um with all of these universities, we were using some legacy code in this

program called Circuit Python. And Circuit Python's great for students because Python um is kind of easy to learn and pick up. But what we were realizing is Circuit Python was a very long legacy code. And because of that, it was getting harder for people to pick up. It was also getting harder to collaborate because the architecture of Circuit Python was not very conductive for building embedded systems,

especially complicated embedded systems like satellites. And so it felt like we were refactoring things kind of over and over as we were trying to add new features. It felt like two steps forward and one step back. And the other thing with circuit Python is um it had a lot of limitations. For example, it had limitations with uh you can only have one thread uh which is a

little bit rough when you're doing a lot of flight critical things on satellite. And so uh with five months until handoff we decided to switch to F-Prime which is an open-source uh componentized flight software framework developed by JPL and Zephyr the real-time operating system also open source developed by the Linux Foundation. Um and this has a lot of things that we want in a satellite. Uh F-Prime

is statically compiled. Zephr is a real-time operating system. There's all of these safety features within it. Also, from a design point of view, not only does F-prime have flight heritage, um flight heritage means it's like flown in space before, that's a very important term in space. If you ever go to like a space guy and they're like, "Our tool has flight heritage." And so they're really happy

about about that because you can they can sell it easier. Um, and so a lot of the tooling that F-Prime has component, uh, it's a component-based architecture, uh, which means that you're able to kind of split off the individual things you're developing in F-Prime into these small things called components. It makes it really easy to test and communicate and debug. And as we found out, it made

it really easy to plan out what a satellite architecture looked like. Um, so now we're going to talk a little bit about our lean development. So, first I'm going to talk about our iterative minimum viable product. Over here is our real white boy diagram. Um, when we decided to restart our flight software from scratch with five months left before launch, um, and we went and we wrote

out everything that was on the satellite and all of the software that we thought was important to be on the satellite. Um, Starch is going to point some stuff out from us. At the top, we've got our antenna board. Our antenna board has a distance sensor, a temperature sensor, and a light sensor. It also has the antennas which we use to communicate. Uh over in the middle

hidden under the evil face is the flight controller board. That's the brain of the satellite. On it we have two radios. A UHF radio which we mainly use and it's what radio amateurs talk to and then an Sband radio uh which is more useful for us for things like high data rate stuff. Uh we also have an SD card, an I2C mug, a watchdog timer, a bunch

of other fun stuff. Uh then we've got a battery board which has a temperature sensor and the batteries um and the battery stuff. And then on each of the corners we've got our XY solar panels. Um those are the solar cells that we soldered 540 of um on on those on those faces. And they also have temperature light sensors and then motor drivers which are the drivers

that allow our satellites to detumble. Um at the bottom that's a hole because we took the payload out but there is usually an XY panel as well. And over there there's a bunch of other things, CDH and comm modules and all of these things that we want for our satellite. And so that's a lot. It's a lot to develop in five months. It's a lot to develop

safely and robustly. It's a lot to make sure it it's all able to run together. And so as a result, we developed an iterative MVP. What is the minimum thing that can be on the satellite that we would consider a mission success? And we would do that and we would also every week revise our MVP. and we would see if it was still what we thought was

the most important. The big thing when you're doing uh space systems is risk. You have to understand what you are risking based on the decisions you're making because what flight software really does is risk mitigation. And what F-Prime and Zephyr really did for us was also risk mitigation. And so um an example of us kind of pivoting from an iterative MVP is we have a distance sensor

on the antenna deployer and we wanted to use that to make sure that our antennas had deployed successfully. Uh that's important because then we know we can communicate with the satellite pretty reliably. But then when we went to work on the distance sensor, we realized that there was no Zephyr supported drivers. Um and the amount of time that it would have taken to get somebody to completely

rewrite that was not within like the fivemon framework with everything else that we had to do. So we said, okay, what is the risk of removing the distance sensor from the satellite? And what we found was we found two risk mitigation strategies. first um we decided to have a deployment uh process that tried to deploy three times. And so even if it didn't know whether or not

it would deploy, it would try again. Um how we deploy actually is kind of fun. There's a resistor at the very top here and the resistor has a little string attached to it and we turn the burn wire on. The resistor gets really really hot and it melts the wire and then the antennas go like that. Um, and so if we just turn the burn wire on

like three times for an amount of time that we've properly tested, that lowers our risk a considerable amount. The other thing that we talked about is there is a way operationally to save the satellite if um if it doesn't if it doesn't deploy the antennas properly. And that is get a really really big antenna dish and just yell really really loud. And even if the antennas aren't

deployed, then it can still hear the deploy deploy deploy command. And we had like a university we were friends with who had an antenna that would we would potentially borrow. And so we were like, all right, we have two risk mitigation strategies. We know we don't have a lot of time. We're going to table this and we're going to iterate our on our MVP. And that's what

we would do every week. Um so that was a pretty cool lean development thing that we did. Um we also did leveraged development. What this means is we leveraged the frameworks that we used. Um so the first framework that we used F-prime a really awesome framework it's component types like I mentioned. So over here we've got like a component diagram. We have the payload communication component and

then the camera handler component. And those used to be one component um and they used to just handle the camera. And what we said was we have five different payloads and a several of them are using UART and we want to be able to test the UART stuff and like we've also got a multitude of different codes. And so what we did is we separated all of

the UART communication and the handshake protocol into a component that was easy to test. And we told each university to develop their own camera handler that was a very small lightweight component that even if somebody didn't have a lot of software development experience, they could still um they could still like look at what the camera handler looked like and develop something that looked uh pretty similar and

that would work. And that allowed us to have leverage testing. It also allowed us to have the five satellites look as similar as possible. Um, it was also really cool because FP prime comes with a Zephyr UART driver which has been tested and validated. Well, not necessarily on flight, but it we didn't have to develop all of these other components ourselves either. Uh, we leverage Zephyr as

well. Zephyr has like drive like drivers for stuff that already exists within Zephyr. And the really cool thing about that is that we do hardware and software code collaboration. And so, um, when something didn't have great support on Zephyr, our hardware guy would come and he'd be like, I got you. this temperature sensor exists on Zephyr. We can put that on our next version of the flight

controller boards. And I you guys heard we had a lot of versions and so we were able to iterate co-hurrenly on hardware and software. It wasn't like ah hardware is done now this is a software problem. It really was a collaborative process and that made it really easy for us to iterate quickly. Another example is that when we were writing our software we realized that the radio

couldn't reboot um properly after it had gone below a certain voltage. And so talking to the hardware team, they were able to add a proper reboot pin uh which was a problem discovered in software and instead of adding a lot of like additional problems to software, there was just a quick hardware fix done for the next iteration. Um and also an advantage of developing software quickly because

if we hadn't done all of this mode management stuff very early on, we might have not caught it right away and it might have been harder to iterate on the hardware. >> So, now we're going to talk about everybody's favorite thing, testing. As you've already heard, we had a strong collaboration between our software teams and our hardware teams. This is kind of abnormal in industry. Usually, you

get the hardware and then the software people have to fix all of the problems. And well, my friend who works in aerospace software has many fine things to say about that. But in the end, what it comes down to is software fixes the problems in hardware because software is easier to spin than new boards. But when you iterate, you can actually reduce the um overall time, but

it means you need to do realistic testing. If you're not testing on hardware and you're working in the embedded environment, you're probably not really testing. So you can run all the tests you want on your laptop that do the unit tests and prove the code and everything looks good and then you jump on the hardware and realize oh I'm out of space up I tried to use

a pin that didn't exist and the whole thing goes up in smoke hopefully not literally and so as a result on this project we deployed hardware in the loop testing as part of our continuous integration process on every pull request that we did every change set that came in as part of our software we deployed that onto our hardware onto the realistic board. Sometimes this exact box

to make sure that we could test on the actual hardware that was going to space. That shows us that our software runs well in the realistic environment. If you don't do this, you start to run into problems with are there enough resources on this board? Are the assembly pieces of the hardware done correctly? Are you missing things like a wire to reset the radio? And so that

kind of realistic testing is imperative to make sure that your hardware software co-developed solution works correctly. The other thing that we did was realistic insitue testing or at least the best we could. It's kind of expensive to send in here up into space with a multimeter to look at the cube set operating in space and test it to know if we can launch it appropriately. That would

be a very expensive uh testing campaign. However, what we can do is climb to the top of the mountain that is um over there, Mount Wilson, and we can also send a team down to Signal Hill in Long Beach. And that gives us a line of sight, pretty much unobstructed of about 50 miles. So that gets us a realistic test of what this thing might do in

orbit with respect to its communication because we are transmitting over a 50-mi air gap using these antennas and talking to it. That's that image that you see over there. It's several of us standing on top of Mount Wilson looking down at Signal Hill. And since it's the future, they were also on Zoom and we were talking to them as they were actively testing with us. >> Discord,

sorry. It's more of the future than I remember. So, as a result, we could do realistic comms testing without leaving pretty much Southern California, which gives us really good confidence that once our antennas are deployed, we're going to be able to talk to them in space. And there was some concern because some of the drivers that we used from Zephyr didn't give us the same gain settings

that we expected with the chip. Doing this kind of realistic testing here on Earth builds confidence for when you're going to send this thing into space and not have a reset pin, except the one the hardware team added. The other thing that we did that really helped development was streamlining decisions. We invented what we called the rule of two. The idea behind the rule of two is

any two people can make a decision. Should we make a decision here live today? All right. I'm saying I'm getting the yes. The idea is if two people on the project agree on a decision, let it be. So you should be able to trust your team to bring in the right experts who know the right uh things to say and discuss. This is in counterposition to what

I've seen in aerospace software where you get every single person in the room and jam them all together and say let's come to a consensus on a decision. And the first thing that happens is everybody says well I'm on vacation. um you know I'm fetching coffee with my friend next week and so three months later you schedule the meeting and then you try to bring 90 talented

people to consensus. It doesn't work real well. Takes several meetings. So now it's been what 6 months or a year and you're finally getting your decision. Well, if you have two people sit down and say who do we have to involve in this conversation? All right, let's make a decision with just those stakeholders and try it. you've tried a solution within um what a week, a day,

an afternoon, maybe even 30 minutes. That's a lot faster than six months. So rather than striving for the perfection of absolute team consensus, we developed this rule that any two people could make a decision and try it. And as a result, we could iterate much faster. And yeah, the first try might be wrong, but the second try is probably right. And the third try is most definitely

right because you're converging on a solution and so three afternoons rather than three months. This kind of decision streamlining process took what is traditionally a very long time to make decisions and changes and compressed it into a schedule that we could complete >> All right. And now we're going to talk about the lessons that we learned along the way. Um uh so the first lesson is to

plan for changes. Over here you guys can see the proofs uh project flight image and then you can see the proofs fight flight configuration hot fixes and then you can see the final final flight config and then you can see the final cubed flight config which is kind of nice because it is a cubat so maybe it was always meant to be. Um, and so part of

how we mitigated this is that we uh try to have a final code freeze and then a final bug bug freeze where we left a lot of time for bug fixes. Uh, but of course sometimes things would come up uh that we hadn't realized within this last minute testing. So an example of this kind of hot fixes that we looked at is um we had authentication on

our satellite. And an important thing about authentification is you don't want somebody to send a bunch of malformed packets to you and spam the satellite and then you keep getting like the message, oh this is invalid, invalid, invalid, invalid, and it drowns out all of your valid uh packets. That's a like valid attacker uh way. And so what we said was, oh, like after the final code

freeze, we're going to throttle all of these events. So they're only going to happen three times and then after that, like on next boot, they'll happen three times. But if you have the wrong authentification and you didn't see those three messages, there's other ways for you to see like telemetry channels and stuff, but we're not going to prioritize you knowing every single message that is mal formatted.

Um, and so we went to throttle all the events and we realized, oh no, there's these two events that we can't throttle and we can't throttle them because they're in upstream F-prime. They're not in code that we can directly edit. Now we could fork upstream F-prime and then for the rest of time have to continuously maintain our fork of F-prime as related to with F-prime. Um or

what we could do is we could edit this sequence that we have and we had a startup sequence which would happen on startup. Um and that that was in charge of deploying our antennas for example. And if we um do do that uh we can throttle those events specifically on startup and then we don't have to worry about actually editing the inner code uh because we're past

the the code freeze and we really don't want to edit um things that are on the board. Uh especially because we are on a very small memory microcontroller uh and uh our RAM usage was a little worrying. It was fine but we we're not we're not going to play around with the image too much. And because we had planned for we are going to fist bugs things

are going to come up that we don't know when we do configurations we'll discover things. Uh we were able to plan for that. Uh but as you can see we didn't make one oh this is all of our hot fixes immediately. So always know there will always be more hot fixes. Another lesson that we made is practical concessions. So Starch was talking about the rule of two.

Part of the rule of two is sometimes two people will approve something and it will pass all the tests. But there'll be an edge case that you guys haven't considered and that's normal because um if you're not spending like a week thinking of everything like things will will walk past you. But what we said was practical concession. We have to just accept that there will be reversions.

There will be things that are merged in that aren't perfect and we'll just accept those. We'll revert them and we'll work on them on the side and we'll just keep going full full speed ahead with main. And so that was really helpful. For one, it helped us move quicker because instead of like maybe like reviewing something for three weeks or lo talking to a lot of stakeholders,

we're able to identify the problem within the code pretty quickly as people were running different things and then we're able to revert and iterate. So, it's okay to not have everything perfect right away. You just have to make sure you identify and remove problems as they come up. Another practical concession we made is that our integration suite of tests uh that were running on GitHub actions were

being a little bit uh buggy. And I think part of the thing is you have to uh see what the difference is between your testing framework and your actual flight software. And so what was happening is we were having issues with the actual testing framework. And so debugging that especially really close to integration we didn't deem was the best use of our time because we really had

to close out a lot of things. And so what we ended up doing is we mitigated risk by other ways by like making manual tests um by by running manual tests by running local tests the same integration tests locally and by ensuring that we validated everything that we did. But we moved away from those automated tests which when they make your life better they really do make

your life better. But if they're making your life harder then you got to pivot and find a way to make your life as good while still mitigating >> Do you have the video pulled up? So the next thing we we learned was beware of state. I feel like every time I give a software talk I'm giving this particular um lesson learned which is state can be dangerous.

In this case we wanted to show a quick demonstration of what happens when you aren't obeying or or aren't recognizing the fact that you have uh state to contend with. So in this particular scene that we will show in just a moment, we have a spacecraft and where it is is in the integration test lab. So you can see that picture that keeps um jumping up there

and that picture. There we go. That picture is in the lab when we're delivering the spacecraft to the launch provider. So we're ready to go up and go to space. Now on all of our satellites, we have a notion of what is time? What does time look like right now? Because when you send telemetry channels or you say, you know, software logs, hey, this happened, hey, this

happened, you want to know what time it is. If you don't know what time it is, how can you tell how things relate together? Now, the problem with time is when this thing is up in space, you have to come up with a scheme for how time relates from the ground to the space. Otherwise, you'll be lost in Is this going to work? >> I think The

Wi-Fi is a little bit rough in this round, but I'll keep this open on the side. >> All right. So, we'll we'll just proceed without it. So, we had a video and what happens in this video, if you try to hit play, maybe it'll work. Um, >> it doesn't format the video at all. >> Okay. Well, what happens in this video is you'll see us integrating during

the test and then surprise surprise, the door of the payload opens in the middle of the integration bay. This is bad. Why is it bad? Because the door of the payload is not supposed to deploy until you get to space. It's not supposed to open in the integration lab. Let's just move on. And so, as a result, we had an issue with state. What happened was they

were doing the integration and they forgot that this particular spacecraft never had time set. So its state was something like 1970 or um actually I think it was 2000. And then when they went in and did the integration and set the time for launch, what happens? The door go >> the door >> pops open. Oh no. And so what we had is we didn't reckon with the

state of the spacecraft and it jumped 26 years in the future and it said 26 years is longer than the time I was supposed to wait to deploy the door. And as a result the door opened in the lab, scared everyone. Made our launch provider say are you sure you want to launch this into space? and we said yes because we know what the state is of

the spacecraft and we can tell you that we just jumped it 26 years in time and it will not happen again until we're up in orbit. The other thing that we needed to be aware of is the trust we had in the team. This was one of the biggest lessons we learned on the project. You have to trust your team. We have another little tidbit about hidden

state here, but you can see the various team members around here. And when we invested trust via the rule of two, via um reviews, via discussing with people, we powered through software faster than I've ever seen working in traditional aerospace because we let people be empowered to solve their problems and get space uh craft software that works. The perfect example of this was the day before delivery

of our software and delivery of the satellite. We're sitting in a lab in Kalpali and the Texas team is sitting in a lab in Texas and we're turning around and we get a report from them saying this one cube set is going into a reboot loop and it just keeps rebooting over and over and over again. And so we tried to debug it and after about 15

minutes we concluded that only applies to one cubat. So, we're going to go ahead and move on and finish testing and get our tests done so that we can prove this thing will work, meet our launch deadline, and get into space, right? But we trusted the team and the Texas people had a different decision. They said, "We're going to keep moving on this particular issue because we

think it's critical. We're seeing it on our engineering hardware." And so, they kept testing long through the day. We kept testing along through the day. And then, lo and behold, late at midnight, we popped this giant cannon of confetti cuz we delivered our software. We went home. We arrived for the day of the delivery on Sunday and we had an pull request from the Texas team. Okay.

They're giving us a pull request after the code freeze. And we look at the pull request and we say, "Oh, oh, that line there that would kill our our cubes." Yeah. So, what they had done is they had tested all day in Texas and figured out what commit um caused this reboot loop. And when they sent us a pull request saying, "Hey, this is the thing we

should take out." We looked at it and said, "That's a cube killer bug that will destroy every fi every one of these five satellite in orbit." What had happened is a little hidden piece of state. When the cube set goes to sleep in safe mode and wakes back up later, it will wake up in safe mode. Safe mode triggers a sequence on boot that says, "Let's safe

the radio. make sure everything is in a quiescent state in the most vanilla boring configuration so that we can talk to the ground. That sequence that tells the radio to go to safe ran before the sequence engine was initialized and therefore the sequence engine said uh you can't run a sequence before we initialized. We should go to safe mode and reboot. Aha boot loop go to safe

mode reboot. Go to safe mode reboot. And so any spacecraft that rebooted in safe mode would be permanently stuck in space mo uh in safe mode in orbit. By trusting the team, the Texas team enabled us to get our tests done and prove that everything else on this software worked while also we trusted the Texas team enabling them to go through and figure out this cube killer

bug. And the end result was well- tested software that met all of the checkboxes, improved all of the functionality, and the cube killer bug was removed in one of those releases that you saw on the first slide. By trusting the team, we solved both problems and let people self- select where they wanted to apply their expertise so that we converged on a working solution. So that brings

us to the end of the talk. If you're interested in contributing, come chat with us. We'll be around scale. We're happy to talk about this and we have hopefully plenty of time for questions. We do love questions so please ask some >> questions. Yes. >> Uh what do you actually do to deumble your cubes? Yeah, we have a D dot D B dot dtumble algorithm that was

developed in Texas state. Um, and we use our magnetoers for that as well as our IMU and our magnetometers. >> Talk about what magnetometers are. Okay. >> Did you mention um to to detumble them are using gyroscopes or motors or >> IMU? >> IMU. Oh, what an IMU. Go ahead. An IMU is an inertial measurement unit that gives you roll, pitch, and yaw and usually in several

sets. So you have a magnetic reading and a gyroscopic reading etc. You take that information and then you take the information uh and then you power your magnetokers which are just spindles of wire here in the spacecraft and they produce a magnetic field. So if you modulate that magnetic field with respect to the magnetic field of Earth, you can effectively break the spin of the spacecraft so

that it's spinning and then it just kind of slows down into a stable pointing direction. >> I had a question on your flight software. So let's say you cut a release on your flight software um and you tested against your hit. Did you automatically upload the software on the Hiddle or were those like manually uploaded? >> So yes, we do automatically upload each pull request to the

hardware in the loop testing. So it wasn't just cutting a release. It was any change that came into the software was pulled into our CI system, run on a custom runner that was wired to one of these satellites up until the point in the mission where we had to stop the integration tests because the test harness itself had um broken down and then we started manually running

them. Either way, we tested in >> Yeah. So over here is an example of every single time someone pushes to a PR. So even if it's not like about to be merged in, uh the tests will run. Um, and over here we have our integration instructions, our llinter, our unit tests are built, and our integration tests. Um, the integration test didn't specifically run on this guy, but

I think it ran on on this guy. Yeah. And so here's our integration test uh suite. And then here's all of our tests. So we've got antenna deployer, burn wire. All of these are running like physically on a satellite unit. Maybe I missed this, but um what altitude are you guys uh putting the headlights and what's the lifetime of how long they're going to be in space?

>> So, we're uh launching out of the ISS. So, we're at ISS orbit, which I want to say is like 473 kilometers, but don't quote me on that. Uh wherever the ISS is right now. And uh our orbital lifetimes right now there's a lot of like solar flares and solar activity. So we calculated I think between six months and a year uh the satellites with like the

lead ballast the lead ballast was put to increase the flight lifetime by like 18 months because of kind of the weight on the satellites. Unfortunately uh we had to remove the lead ballast because when we integrated with the satellite the ballast was so heavy. Uh we basically we hit the maximum weight that you could for the satellite because we like oh we might as well go to

the maximum. And it turns out maybe nobody's ever done that because um in the deployer they check how many newtons of force you push. The deployer is like a spring and you put these cubes in a box with a spring. It's a really expensive box with a spring. Um and so they wanted to make sure that the spring will be able to push your satellite out. And

so the newtons of force were very strong because the light is heavy and we're we're in gravity. Um and so it maybe would have worked on orbit, but we didn't want to risk it. Um, and so yeah, >> and that's actually one of the things that I found really interesting. Sorry, I know you have a question. I didn't realize this, but the heavier your satellite, the longer

it can orbit because up there, it's all about the uh resistance, the friction that slows you down. And the heavier you are, the more momentum you have. So the longer it takes for this kind of background friction from solar flares and various other things to slow you down and de-orbit you. So this thing without the lead would de-orbit a lot faster than the original version that was

a little bit too heavy for me to pick up with one hand. >> Uh you mentioned that you were using SD cards. I was just wondering how you kind of determine and calculate that the failure rate of something like an SD card is suitable for your use case and also do you have any kind of like failure mitigations for uh something like an SD card? basically hardware

that's not really in your control your >> So I I would like to clarify a little bit in this particular cube set I don't believe we have an actual SD card. We have a flash chip that has an SD card controller on the front. So from the perspective of software it is an SD card but from the perspective of hardware it's an integrated circuit which reduces the

risk. That's an excellent question. You mentioned there's a way to request a joke from the satellite. Is there documentation on that somewhere? >> Uh there will be uh keep an eye out on our GitHub or on our website the proofs kit proofs core reference. Uh we're going to make a tool for radio amateurs so that you can send a fully formed packet up to the satellite. Um

and our launch is on April 8th. Our deployment will be sometime in May and then after that we'll have the ability to ping the I'm glad you asked that because I have a follow-up question to more comm stuff. Um, well, my first question is about the payload. Um, so you mentioned you also have an Sband um, payload or one of the the cubats had one. Um, is

that something you were able to test with um, like a ham radio um, license as well or um, okay. Um so um do are you are are you able to use the same ground antennas or did you have to um subscribe to to other other stations? Um was there a separate process? Yeah. So a lot of our team does have like regular amateur license. Oh yeah. Or

is that you have more question? >> Yeah. Yeah. Um so for the um my next question is about the Zephyr FP prime integration. um is that um is that still um actively maintained and supported or um is is there a contact for that? Okay, so I'll answer the ground station and then he'll answer F prime Zephr. So um a lot of our team does have like amateur

radio licenses so we're able to operate stuff. Uh what we would often do during testing is we'd have two flight boards and we'd use one as a ground station. So we had a ground station path through and then we would have the other satellites and uh be the satellite. And so that's how we would test. Right now we're in the process of finalizing our ground station. So

the Texas state's team is going to have a ground station. Uh the UC Santa Cruz team is going to have a ground station. Uh we're friends with the Hawaii team who who said we can also use their ground station. And so we have like university partners that we're going to be operating the satellites from. And so those are going to have like SPAN capabilities and UHF uh

Laura capabilities for our Laura radio. Um, we are prioritizing Laura UHF because that's most of our operational guide. Our satellites right now actually don't have Sband capabilities even though we have a Sband radio uh because right before integration we're running into issues with dynamic memory allocation which is a big no no uh with the Sband radio. And so those are going to be uh on the next

satellites that are being handed off in May and launched in in August. Those are going to hopefully have the Sband radio capabilities. And then as for F-Prime Zephyr, F-prime Zephyr is a F-Prime community package that brings in Zephr arts support. It is being actively maintained by several members of the community. Uh one of them being the open source space foundation, one of them being the F-Prime team

and one of them being industry partners who are also running on top of F-prime Zephyr. So right now F-prime Zephyr is actually a fairly stable uh art toss I think you might have hinted to this, but um how are you securing communications to the satellite? >> Yeah. So, we have a CCSDS compliant HMAC communication protocol. And so, we're sending up um security headers and footers that have

a sequencing number to prevent repeat attacks. So first the satellite will check if the sequencing number has incremented so no one's repeating commands and then it will check an HMAC validation against the key that is stored securely on the satellite and only authenticated packets will pass through uh the comst stack unless they're like a few ones that are reserved for radio amateurs such as the tell joke

command. The tell command can pass even when it's not authenticated. >> So some of you who do cyber security may note that isn't a full cyber security protocol. We did a risk analysis of the satellite and determined that really the most dangerous thing somebody could do with this satellite in terms of the larger space community was to update new software and use the magnettors to kind of

pilot it through the the magnetic field. Very unrealistic, but we did make sure that the images that run are not something that somebody could update without having access to a separate imaging key. And so the images themselves are locked down. The authentication prevents unauthorized access, but the overall communication is still visible to the public. They just can't command or update software. >> All right. So my question

is um since you said that the ground station is still under active development, is that going to be open source if I want to have a satellite tracking ground station? >> Absolutely. Um, and there are also already open source ground stations. Uh, if you check out satnogs, uh, they've got a lot of resources for that as well. >> Okay. And then quick small second question. Are the

jokes out of this world in a less literal way? Are they really that funny? >> Um, one of the jokes may or may not be uh, the satellite is out of this world. I can either confirm or deny. You can though because the jokes are also open source and the jokes are on GitHub. And actually, if you go to our codebase, you can make a poll request

to the jokes file and add a joke there, and I will probably approve you if your joke is funny enough. Um, and maybe one of your jokes will be down linked from space. So, check out the proofs core reference. There's a QR code right there. >> Yep. Uh, so you mentioned about the burn resistor as being a fun hardware hack. Could you say a few words about

the measuring tapes? And do you have any other fun hardware hacks? >> Measuring tape. Okay, so this is something that gets me really excited because I'm not really a space guy. I'm a software guy, which is why when you asked how high we are in orbit, I'm just like, "Yeah, that number sounds right cuz I do software." But the measuring tapes. Okay, so it turns out that

the kind of antenna that you want on uh cubats are pretty much a metal bar that is about the uh width and length of a measuring tape. But the measuring tape has a really useful property in that it springs back like that. And so many many lowerbudget cube set teams just use measuring tapes. And if you come up here and look at this the spacecraft uh a

little bit later if I don't destroy it entirely, you can look in there and all we did was uh um sand off well you sanded off the uh paint on the measuring tape where it connects and otherwise we just left the measuring tapes in there because this is iconic. uh lower budget cubat development. >> Yeah. Some I think some other or maybe we should keep going. Uh

some other fun hardware hacks are our frame. A lot of people will do like a machine frame which is very expensive. Uh we did not do that. Um but we kind of had a jig. We had a 3D printed jig that we measured that was super accurate. And so when we ordered our frame, we were able to like measure against it. And so instead of having like

a machine shop be very precise, we just ordered a bunch um and measured and kind of like screwed them in uh because this is a little bit flexible when it's not properly screwed in um in order to get the right size. Um and yeah, we have we have a bunch of other fun fun hacks as well we can talk about later. >> You have five satellites and

I think five teams. Why not share satellites? And uh what was the reasoning behind each individually having your own? Was there separate payloads or what's going on there? >> Yeah. So, the first thing is that this this kind of mission happened uh because we've got an open source kit. And I'm sure you guys are big big on the open source community. At least I really hope so.

Um and one of the things you guys surely have encountered with an open source is there's open source and then there's open source with good documentation and that people can recreate it and copy it and actually learn from it. And the proof kit kalpoly pom had been developing for a while. They launched many satellites, but they wanted to take the next step and say we want people

to use it and we want to make sure they can use it. And so by saying each university is going to individually make this open source kit, we can see how different stakeholders and different people are able to read this documentation, contribute to it and like have different um aspects. The other thing was we were also geographically separated. Uh five universities in pretty different parts of the

US. So it would have been really hard to make one satellite together. Um, and these kits were also designed for a oneU. We just added the 2U, which is the unit of measurement. The small ones are one UU's. And so we got like six U's of space to launch satellites. And if we've got six of space, we're going to use all of them. And finally, I think

I mentioned this a little bit earlier, but each of these satellites have like a different payload. Uh, curious to piggyback right a little bit, you have all of these different people with different skills, I'm guessing, or or interests or subin interests. How how is it like as a leader when you're like, "Okay, we've got to build this thing." How do you determine, oh, this is the best

configuration for us to work together? >> Yeah, that's that's a really really good question and it was something that was really cool to find out in a team that was so asynchronous. And I think a big part of it is kind of this last slide that we were talking about, which is like trust the team. We make big decisions together. we had like all of these plans

and stuff, but when we give somebody a task, we trust that they can accomplish it, that they can design it, that they'll document it in a way that we'll be able to understand, and they make decisions that work for the satellite. And so, yeah, and so we were able to kind of iterate quickly and allow people to make decisions. And there's many ways to to make a

satellite. Um, but we were able to find a way that worked for us and our team. And I just wanted to plug here because this is something that slipped my mind when we were talking about trusting the team. I'm a fairly seasoned flight software developer in aerospace. That's my day job. I'm not going to talk about it much because I'm here behalf of open source space foundation.

But trusting the team and trusting the leadership meant that I had to step into a role of not really being a leader but being a follower because I didn't have the time to volunteer for this apart from a few evenings and weekends. And so when I looked at the team, I was ready to be the leader because guess what? That's the role I normally take on. And

then I noticed other people and as uh doing the leadership role and I'm like, "Oh, oh, they've got this. I can just let this one go." Which was a lot of fun for me. But it also made the team stronger because we let those people who are natural leaders and knew what they could contribute and how to orchestrate these people come together. So letting people step up

even if you have a higher experience level than them gives them an opportunity but also makes for a better >> Uh question. So real quick as someone who's worked in cubat projects both uni and professionally um I empathize with you folks who have uh stayed up late nights um eating leftover pizza and what have you just to do testing. So kudos. um so you've mentioned the value

of open source development uh componentization uh things like importance of iterate development. Um I even noticed in one of your slides you're using what appears to be semantic versioning for your code releases. Is that right? Um so the these are all uh practices and uh uh approaches that like are very uh adaptable in software development, have you applied similar approaches uh to hardware and if so what

kind of pain points have you experienced? I think we mentioned it a little bit, but like to expand on it, um I really think like our lean development really came from the fact that the proofs kit was developed to be a lean hardware kit. Um we're a very very cheap satellite when it comes to um cubats and that was because the hardware was designed to be iterated

on quickly and was designed to be affordable for universities um and manufacturable. And so we went through a lot of designs of the hardware throughout even these last five months because the hardware team was really leveraging these lessons of lean development. Um and so I I would say like definitely that was like an important thing for for us and something we learned from the hardware team. >>

So sorry to clarify the speaking more to the effect. So you probably share software and open to repeat the question because uh the microphone wandered further back into the crowd. Do we publish the uh hardware stuff as open source? The answer is yes. I believe ENS is showing us right now. And what kind of uh approaches did we take to hardware development? So I don't think either

of us are are entrenched in the hardware team. I'll take a guess. Our hardware people did do revisions. They did do GitHub development and they did check-ins and that sort of thing. And then I believe we've published our design schematics the Kyad files or or something similar. And Kyad is >> Kad is it? Oh, Keycad maybe is what they were trying to say. Okay, I'm a mess.

Let's move past this. Uh, but it's available and is the documentation. And so, theoretically, you can go to here, check it out, fab your own boards, build it, upload our software, and fly it if you can get a ride to space. Does that kind of answer the question? Okay, cool. Um, so you showed us that video of the gate swinging open and that was a like an

error resulting from some spacecraft state um and um just not having a set handler for time on Zephyr. Is that something that is um um intentionally um restricted in Zephyr? And um is now a good time to ask about like how that was resolved? Oh, so the issue was when you boot up our flight controller, we didn't have I don't believe we have a real-time clock on

here. And so it boots up in the year 2000. Zephyr does have a set time handler so that you can sync time. The problem was we booted up in one state. We never Oh, we do have a real-time clock. We had never set the real-time clock before. So the time was 2000. It knew the time was 2,000. It tracked the state was 2,000. Then we updated the

time and it was a delta of 26 years which was longer than the 45 minute wait for deploying that door. And so it wasn't an issue of that stuff not being supported. It was the fact that we didn't preset the time before going into integration and so we had this large time delta. So what is the solution? The solution is when you first boot up the board

and first run the RTC, just set the current time and then from then on out, you'll always have a reasonably close time modulo the clock drift of your real-time clock. Does that answer your question? >> Uh, thanks for a great talk by the way. Um, question or two about uh power and thermal issues. Uh, does it have a battery or does it just turn off for 45

minutes every other 45 minutes? And uh, also maybe you could talk about do you have to worry about this thing overheating? So, it does have a battery. If you uh, ENS here will show you there's a four cell battery uh, board deep in there. And so, as it orbits, it will charge the battery from the outer planes. Now that being said, one common problem with cube sets

of this size, I'm not going to say specifically for proves because ENS can fill in the details, but power management is difficult because you only get so much power in from the solar cells. And so you charge your battery, but you should expect when you're orbiting in space to have some amount of downtime or quiet time where you can wait to charge. And then thermal management is

another issue. In space, you don't have this atmosphere to conduct space uh uh air uh conduct heat away from your spacecraft. So, you do have to manage both. Uh ENS will tell you more of the details on how we approach that. >> Oh, yeah. And then also really quickly, I think you mentioned like does it like sleep for 45 minutes? Our satellites don't ever sleep, but what

we do for safe mode is that we turn off all the peripherals. And so, all of the sensors are off. Um a lot of things that would gain power are off. the payload is off and the only thing that is on is the radio and it's just sending like health packets at a very like low rate so that we can conserve as much power as possible. And

basically the idea is if an anomaly happens we want minimal systems running because who knows what that anomaly is from. So that 45 minute wait is just like like just hang out in the satellite and don't do anything until the 45minute wait, but the satellite's still on and it's running. And then after the deploy, we'll turn the radio on and we'll deploy the antennas and all of

that fun stuff. And then once the radio is on, then we'll start like sending out packets. Um, and then in terms of the the the heat um things, some satellites will have battery heaters, some satellites will have heat sinks. Because our satellites are pretty simple and what they're doing and they're not emitting a ton of heat. Um, but also we like calculate that we didn't need battery

heaters specifically for our battery, we actually don't have any active um thermal components on our satellites. Uh, just passive based on kind of the satellite design. In the past, the proofs kit has had battery heaters. Um, but the thermal team decided that wasn't uh essential for this uh launch. And keeping it simple is pretty important for us. >> One more thing. >> So, one more thing, the

uh one place that cubets run into a lot of power and thermal issues is when they ship a big computer like something capable of running Linux. We've got an RP2350 in here. So, it's a much smaller, much deeper embedded processor and a whole lot smaller to work with, which is why you've heard we had some RAM problems that we alluded to. But the importance of that small

processor is it keeps the heat output and the power consumption of your primary computer very small. One thing you might do and I see a lot of cube sets do in the field is they have a small computer like the RP2350 that runs these satellites and then they have a payload computer that they can turn on and off when they want to do much more compute intensive

things. We don't have that in the proves design because it's not required to meet our mission parameters but that is another option in the space. It sounds like at the end of your talk, uh, you guys were kind of down to the wire and came across a boot loop from Texas Tech or University of Texas. uh uh I was wondering how that didn't get caught in your

hardware in the loop tests and if there was any key takeaways on how you structure your tests or how you would structure your tests in the future from realizing that that wasn't caught or was it caught? I don't know. >> Thank you for asking that question because Enes told me I should talk about that during the slide and it slipped my mind. So why didn't we go

into safe mode in any of our tests? The answer is we did go into safe mode. The problem was we never bothered to upload the final safe mode sequence. And so we would go into safe mode but never try to run anything and therefore not invoke this problem. So what recommendation can I give you about doing better tests? Make sure that your system is in a complete

state and upload all the necessary files even if they're placeholders or reasonably small so that you have as close to flight as expected. >> Another limitation of our integration tests is that if we ever did like a reboot during integration tests, uh the connection to the integration tests would fail and it'd be really hard to automatically have it like catch it back up again. And so we

didn't have things like, oh, what happens when this anomaly loops just because of the limitations of our testing software. We also didn't have tests that like run everything on the sideline at once. We reserved all of those tests for like integration testing at the end and like operational testing because that was just easier. But definitely a lesson learned is to do stuff like that, those states that

you're not testing into integration tests way more often. >> Yeah, I have two what I think are quick questions. What happens at the end of the life cycle of the cube? Like are they reusable or does it get burned up in the atmosphere? What is happening to it? And uh how do I get one of those cute little plushies? so uh the FCC says that your satellite

has to de-orbit within five years for like a regular satellite license. Most cube sets don't need like propulsion or anything to do that because they're so small and they don't have enough momentum. And so within five years, they'll just go slower and lower and lower until they burn up peacefully into the atmosphere. And so that's what's going to happen to all of our satellites um well within

two years because of the weight and just this the solar flare activity level right now. Um and then for the cubat plushies um I think there's a website called like makeyouplushy.com or something and um our systems lead he sent a picture of one of our one use to that website. You can draw like a anything and they'll turn it into a plushie. So, they turned our satellite

into a plushie and then they ordered a bunch. We are taking pre-orders. I think we ordered like a few hundred or hundred or something and they all sold out. And so, now in order to do like another order, we have to get a bunch of pre-orders because it's kind of expensive cuz they're like $40 each and if we're ordering like a hundred, it's like several thousand dollars.

But if you're interested, I'll add you on to the list and hopefully within like a year or something, we'll have enough people for another order. So the satellite will fall down within five years, but you you said it will last maybe to a year from the friction. Will it be operational through that whole year or will the batteries die during that time or what what you expect

it to run until it falls down? Yes, we believe it will run until it falls down. Modulo space is hard and so things can happen. Solar flares, all sorts of stuff. we could get hit by a piece of space junk. And to answer your earlier question that you asked me in the hall, these aren't going to become space junk because they will de-orbit and burn up. And

the reason that the Cubat licenses have these durations to de-orbit is because the amount of stuff in space is growing. And NASA is actively trying to push for a model where things have a timeline to de-orbit to keep our skies clutter-free. Um, and also in terms of like uh losing power, that's partly why we have safe mode. So if the satellite is like draining power, the solar

panels will hopefully become power positive and recharge the satellite. And so we'll turn everything off to like make sure the satellite is able to get to like a healthy power. >> Thank you for your wonderful questions and thank you for this wonderful presentation. Thank you. If you guys want to come up and touch the satellite, be very careful, but we can we can have that. Um, and

if you guys have any more questions, >> don't kick it like I did. >> Don't kick it.