WEBVTT

1
00:01:29.890 --> 00:01:34.329
senate chamber: Corporation. By the way, when we finish the recording absolutely.

2
00:01:52.610 --> 00:01:57.880
senate chamber: So I'll give you a general intro that Mike is the session chair. He'll introduce you and

3
00:02:05.400 --> 00:02:06.080
senate chamber: minutes.

4
00:02:07.170 --> 00:02:08.980
senate chamber: Do you want me to leave the slides up? Or

5
00:02:09.419 --> 00:02:14.979
senate chamber: oh, yeah, you can. Yeah, no, let let's yeah. Let's let's swap back to the view.

6
00:03:06.250 --> 00:03:06.970
senate chamber: There you go

7
00:03:22.240 --> 00:03:30.269
senate chamber: alright. So let's get started on time just to check. Can people online hear us? And can someone say something?

8
00:03:30.790 --> 00:03:31.443
senate chamber: We're good?

9
00:03:32.160 --> 00:03:37.080
senate chamber: We can want them to say something, so to make sure we can hear them.

10
00:03:37.460 --> 00:03:39.160
senate chamber: Andrea say something.

11
00:03:39.530 --> 00:03:40.670
Andrea Stocco: Something.

12
00:03:40.670 --> 00:03:42.700
senate chamber: Yes, very good. Thank you.

13
00:03:42.920 --> 00:03:56.270
senate chamber: Well, welcome to the 30 second actor workshop power of 2. It's going to take a while before we hit the next one. We have a nice in-person audience. We're still hybrid.

14
00:03:58.950 --> 00:04:25.129
senate chamber: a few logistical announcements. 1st next door. There are more stressful people than us taking the Ohio State Bar exam. So they asked us if you're going to the bathroom, come out this door, turn right and go to that side of the building. Don't go back to where you came from, because there are people there who will escort you to the bathroom and make sure you're not cheating with your phone when you go there, so you may not want to do that.

15
00:04:27.850 --> 00:04:34.000
senate chamber: Another announcement, the next Iccm

16
00:04:34.210 --> 00:04:43.239
senate chamber: will be next year in Montreal. The tentative dates are July 17th to the 21, st

17
00:04:43.370 --> 00:04:51.440
senate chamber: Neley, do you have any further announcement to make? Beyond that, that's that

18
00:04:54.180 --> 00:05:12.589
senate chamber: protocol for connection. So everybody, whether that's main speakers or panelists. If they have slide panelists can just make off the cuff remarks. But if you have slide, you're sharing through. Zoom. You're connecting to zoom you're sharing through. Zoom.

19
00:05:12.590 --> 00:05:25.649
senate chamber: Make sure your microphone is muted and your speakers are off. If you're in the room. Nobody is switching to the connection laptop. It was hard enough to get the presentation laptop set up the right way.

20
00:05:29.110 --> 00:05:54.019
senate chamber: otherwise. So the we're doing something a little different this year, instead of having a bunch of presentation around loosely defined themes, we decided to take a longer look at the future of cognitive architectures. Among other things, with everything that's happening around. Generative AI seemed like a good time to take stock of where we are and where we go.

21
00:05:54.020 --> 00:06:12.759
senate chamber: so we'll have 4 sessions. The 1st sessions will be, and the format of each session is essentially a 20 min presentation to set up the groundwork. This is meant to be very discussion heavy. So the 20 min presentation to set up the background for everyone.

22
00:06:12.810 --> 00:06:32.749
senate chamber: Then we'll have a panel. Each panelist has time to make some remarks no more than 5 min, please. I will be timing. So when the main presentation is over, panelists should come over here and sit there, so everybody on Zoom land can see them, and either present their.

23
00:06:32.750 --> 00:06:35.430
Yang chengta: Or make no remarks.

24
00:06:35.430 --> 00:07:02.339
senate chamber: Sitting here for the rest of the discussion, session after panel statement that everybody should ask questions. People in Zoom Land can use the chat. For that matter. People here can use the chat to discuss various things, but when we get to the open discussion session, please raise your sort of your virtual hand, and we'll try to manage the in-person questions and the zoom questions and make sure everybody gets a reasonable turn.

25
00:07:03.080 --> 00:07:28.450
senate chamber: So, to get back to the program. We'll start with a presentation by Joe Harpt and Utilia Larue of a recent paper. They had using mathematical psychology techniques to evaluate the assumptions of the Aktar theory, and I think that that's an interesting and provocative paper to tell us where we're falling short and maybe suggest more things we should do, how we should remedy those issues.

26
00:07:28.790 --> 00:07:35.330
senate chamber: The the second session will be about sort of one of the elephants in the room

27
00:07:41.400 --> 00:07:55.029
senate chamber: model of cognition, and what we think about it, how it compares to what we're doing, and what it might suggest about what we want to do and what kind of standards and benchmarks we want to meet

28
00:07:55.220 --> 00:08:15.209
senate chamber: the 3rd session sort of following. That will be about potential ways of hybridizing or combining cognitive models, cognitive architectures, and generative models. There are a lot of ways. It's a very rich space to explore. And so we get to talk about that.

29
00:08:15.210 --> 00:08:42.809
senate chamber: And then the final session is about an oncoming effort funded by Nsf to develop an open source ecosystem for the actar architecture. It's finally addressing those issues sort of like the development of the software and to some extent the management of the community has been fairly ad hoc. And I think it's way past time that we take a more systematic view of that.

30
00:08:43.039 --> 00:08:53.670
senate chamber: And it's worth sort of keeping that long range perspective for actar in mind when we go through the previous sessions. And we we discuss those topics.

31
00:08:54.250 --> 00:08:57.619
senate chamber: So I think I'm done with my

32
00:08:58.650 --> 00:09:04.740
senate chamber: I think I'm done with my remarks, so I'll hand in to the session chair for the 1st session. Who is Mike Byrne.

33
00:09:11.350 --> 00:09:18.210
senate chamber: And I'm going to be really brief, because we have a lot to cover this session. So I'm just going to let Joe take it away.

34
00:09:32.620 --> 00:09:39.300
senate chamber: So maybe so. First, st I want to just thank thank you all for inviting me. I'm very excited to participate in this workshop.

35
00:09:40.050 --> 00:09:47.520
senate chamber: I think the cross fertilization of ideas from the math psych tradition of cognitive modeling. And the Rccm tradition.

36
00:09:47.790 --> 00:09:54.350
senate chamber: particularly actor, just continues to have a lot of exciting opportunities for doing things in new ways.

37
00:09:55.050 --> 00:10:04.680
senate chamber: As part of that, I would like to say I'm definitely don't. Don't think that my place or my ability even, is to say, you know anybody's doing anything wrong

38
00:10:04.800 --> 00:10:07.749
senate chamber: outside of, you know. Maybe some sft people I can.

39
00:10:08.750 --> 00:10:17.190
senate chamber: but more hoping to demonstrate how, from from my perspective in the Sft world, and and working with Chris Fisher, who.

40
00:10:17.300 --> 00:10:20.499
senate chamber: I think, does an excellent job of of sitting in both worlds.

41
00:10:21.940 --> 00:10:27.194
senate chamber: That. And and you know, in collaboration with Athalia and Kevin that we can

42
00:10:27.820 --> 00:10:30.070
senate chamber: look to see how these tools can work together.

43
00:10:31.284 --> 00:10:38.630
senate chamber: So this particular paper we framed around the concept of global model model analysis.

44
00:10:39.000 --> 00:11:01.369
senate chamber: The broad idea behind global model analysis is to step away from something we sometimes get caught up with in the math psych world of being very heavily focused on parameters and best fit parameters, of how well a very particular pattern of parameters might describe the data and instead, try and talk about the model across the whole range of what it might be able to predict.

45
00:11:02.043 --> 00:11:07.359
senate chamber: And I think for for cognitive architectures as well, the goal is.

46
00:11:07.570 --> 00:11:15.739
senate chamber: in addition to just parameters, the different kind of combination of ideas or theories represented within the constraints of that architecture.

47
00:11:16.940 --> 00:11:24.140
senate chamber: And so the goal, then, is to see if we can take that zoomed out level and start to analyze what? What are the core

48
00:11:24.680 --> 00:11:26.200
senate chamber: guiding principles.

49
00:11:30.380 --> 00:11:38.840
senate chamber: the particular tool for this paper, and one I'm most familiar with. So this sort of sledgehammer

50
00:11:39.311 --> 00:11:41.960
senate chamber: that I brought along is systems, factorial technology.

51
00:11:42.973 --> 00:11:52.340
senate chamber: This approach is framed around analyzing how people combine multiple sources of information for making decisions or actions or

52
00:11:53.730 --> 00:11:55.270
senate chamber: combined percepts.

53
00:11:55.810 --> 00:12:10.249
senate chamber: And the idea of the framework is to break that broad question of how are things combined into a subset of questions or a collection of questions? One. How those sources of information are used. So whether it's a sequential process of, say.

54
00:12:10.560 --> 00:12:18.909
senate chamber: figure out the auditory information and the visual information, or a parallel process, you know, identifying information from multiple modalities at once.

55
00:12:20.310 --> 00:12:28.760
senate chamber: much is enough to respond and forgive me for using mostly perceptual examples. I'm hoping in the discussion we get to broader examples.

56
00:12:29.070 --> 00:12:36.930
senate chamber: you know, if I if I hear an alert, that's enough for me to do something rather than for me. Sitting around needing to see the flashing light and the buzz

57
00:12:38.750 --> 00:12:48.880
senate chamber: Knowledge of one source affects how we process another. So if there's an auditory buzz going off, does that actually facilitate or speed up the process of me? You know.

58
00:12:49.000 --> 00:12:51.230
senate chamber: detecting that flashing light for an alarm.

59
00:12:52.273 --> 00:12:55.520
senate chamber: And then the the last question here

60
00:12:55.640 --> 00:13:07.119
senate chamber: a sort of resource question, as I start to have to deal with more sources of information. How much does that degrade each of the sources of information relative to? If that was all I had to deal with?

61
00:13:11.670 --> 00:13:25.289
senate chamber: so we tend to to frame. So so first, st you know the sort of vocabulary across fields. So we call this architecture. And it definitely means a different thing from architecture in this context.

62
00:13:25.660 --> 00:13:26.815
senate chamber: Architecture,

63
00:13:28.070 --> 00:13:39.939
senate chamber: in this sense, is a little bit more flexible, I think, than architecture that's used in the Iccm term. So somebody might use a serial strategy for a particular task on one case in a parallel strategy. In another case.

64
00:13:40.660 --> 00:13:44.770
senate chamber: and then the sft world, we would just call that architecture, even if it is flexible.

65
00:13:46.460 --> 00:13:53.460
senate chamber: so so parallel information processing of that example, sort of dealing with both sources of information at the same time.

66
00:13:54.745 --> 00:13:57.419
senate chamber: Serial or sequential information processing.

67
00:13:57.790 --> 00:14:07.920
senate chamber: We're often also interested in the special case of co-active processing. So some of the work that in my lab we do in perception is more gestalt and configural perception.

68
00:14:08.100 --> 00:14:17.810
senate chamber: where there's sort of an identity lost of individual parts of the information, and it's more about the configuration of the whole or the combination feeding into the decision process.

69
00:14:18.800 --> 00:14:20.550
senate chamber: They're kind of special case of parallel.

70
00:14:21.830 --> 00:14:26.600
senate chamber: Stopping rule is the term we use for how much is enough to respond

71
00:14:27.374 --> 00:14:31.409
senate chamber: so, and this roughly corresponds to logical gates.

72
00:14:31.560 --> 00:14:36.670
senate chamber: So I might have a parallel process where I'm you know, listening in for the alarm

73
00:14:36.810 --> 00:14:39.760
senate chamber: and looking for a flashing light

74
00:14:40.560 --> 00:14:43.839
senate chamber: happening in parallel, but again exhaustive. I'm going to wait until

75
00:14:44.010 --> 00:14:46.879
senate chamber: both sources of detected both to respond.

76
00:14:47.630 --> 00:14:51.850
senate chamber: and this, of course, can be combined with a sequential process or a parallel process.

77
00:14:55.381 --> 00:15:06.649
senate chamber: A self-terminating process kind of borrowing language from the memory literature this would be. There's some way to stop the search before you're done. Usually when we're just dealing with 2 things we just

78
00:15:07.790 --> 00:15:15.929
senate chamber: connect that with 1st terminating. As soon as I hear the alert. I'm going to do whatever I need to do with it rather than beginning contrast, waiting to see it too.

79
00:15:20.214 --> 00:15:29.099
senate chamber: The the way different sources of information influence each other as distinct from a resource thing. So more about say.

80
00:15:29.400 --> 00:15:41.689
senate chamber: information about the light is giving me information about the tone, or you know, maybe it's a little more interesting in a semantic context if I'm hearing a word being spoken at the same time, I see it being written

81
00:15:42.210 --> 00:15:50.819
senate chamber: that might facilitate back and forth. Or if I see a person's lips moving that's going to facilitate my perception of hearing what they're saying.

82
00:15:51.350 --> 00:15:55.359
senate chamber: So it's more about shared information influencing one another

83
00:15:55.840 --> 00:15:58.379
senate chamber: other than than a resource question.

84
00:16:00.560 --> 00:16:03.858
senate chamber: Which we frame as as workload capacity.

85
00:16:04.760 --> 00:16:15.910
senate chamber: so, broadly speaking, when we're talking about workload capacity resources. We have the 3 categories, limited, unlimited, and super capacity limited capacity being.

86
00:16:16.420 --> 00:16:31.279
senate chamber: If I have a set amount of resources, and I have to divide them up amongst the thing I'm dealing with. So I'm going to be a little less efficient at each individual source and context. So if I'm slower to detect a tone, if I also have to be watching for a light

87
00:16:31.730 --> 00:16:35.410
senate chamber: that we'd characterize as a limited capacity system

88
00:16:35.880 --> 00:16:39.249
senate chamber: unlimited capacity where there's just no change.

89
00:16:39.880 --> 00:16:41.080
senate chamber: So this might be like.

90
00:16:42.280 --> 00:16:45.670
senate chamber: potentially more realistic in a cross middle situation, where.

91
00:16:47.020 --> 00:16:54.580
senate chamber: even if I have to listen to a bunch of auditory information, it's not slowing down my ability to detect visual information.

92
00:16:56.370 --> 00:16:58.940
senate chamber: The super capacity situation

93
00:16:59.110 --> 00:17:07.131
senate chamber: tends to be, I think, a little more theoretical, theoretically controversial. We certainly, in our measures see this kind of thing show up.

94
00:17:07.700 --> 00:17:17.470
senate chamber: go to example. That that tends to at least get a little bit of nods is more like a social facilitation type of thing. You've got one person doing their job on their own or running the race on their own.

95
00:17:17.599 --> 00:17:22.780
senate chamber: You put another person in there. They're both going to run a little faster. So adding more to the system, kind of

96
00:17:23.140 --> 00:17:26.670
senate chamber: increases the the effort or resources available.

97
00:17:31.160 --> 00:17:43.410
senate chamber: So so very high level, the system. Factorial technology kind of theoretical background and framework breaks down these questions of architecture. And again, meaning more like structure of information, flow, not

98
00:17:43.620 --> 00:17:47.800
senate chamber: fixed structure of the cognitive system, the stopping rule.

99
00:17:47.940 --> 00:17:56.150
senate chamber: So, architecture being parallel, serial, or coactive, and then more complicated combinations of those things as you get more and more sources of information

100
00:17:56.290 --> 00:17:59.539
senate chamber: stopping rule, how much information is enough to respond

101
00:18:00.310 --> 00:18:08.530
senate chamber: capacity. This sort of resource question, as I add more for the system to deal with, how efficiently. Each of those individual sources are handled.

102
00:18:09.260 --> 00:18:15.330
senate chamber: and then dependence across the information sources. So the extent to which information might be shared or sped up.

103
00:18:20.370 --> 00:18:24.679
senate chamber: I guess I don't have a transition slide, but switching over to measures. Now.

104
00:18:24.880 --> 00:18:33.520
senate chamber: one of the strengths of systems, factorial technology is connecting some of these questions to empirically testable frameworks.

105
00:18:34.020 --> 00:18:39.270
senate chamber: So the first, st the 1st tool is, we call the survivor interaction. Contrast

106
00:18:41.690 --> 00:18:47.660
senate chamber: and the idea here is to be able to distinguish among these types of combinations of

107
00:18:47.770 --> 00:18:50.669
senate chamber: information flow and decision rules?

108
00:18:51.920 --> 00:18:57.480
senate chamber: So where we would expect a different prediction empirically out of this kind of system.

109
00:18:57.650 --> 00:19:00.050
senate chamber: as distinct from this kind of system

110
00:19:00.200 --> 00:19:03.950
senate chamber: that is not affected by questions like resource, limitation.

111
00:19:06.390 --> 00:19:11.700
senate chamber: So one of the examples, the actual, the original example, these were

112
00:19:11.860 --> 00:19:15.759
senate chamber: rather than being above and below, right and left visual field.

113
00:19:15.910 --> 00:19:18.530
senate chamber: The idea to come to test do we, in fact, use

114
00:19:19.170 --> 00:19:27.450
senate chamber: left and right parts of the visual field in parallel or proactively the particular data set good

115
00:19:27.810 --> 00:19:30.740
senate chamber: that this one was based on. And they were up down.

116
00:19:30.930 --> 00:19:41.599
senate chamber: So the question would be, if I'm if I've got a a light I need to attend to, or a dot that'll appear on the screen that I need to attend to. That's on the top half of the screen and the bottom half of the screen.

117
00:19:42.150 --> 00:19:46.879
senate chamber: I can ask these questions of, am I dealing with those 2 parts of the screen in parallel

118
00:19:47.010 --> 00:19:51.059
senate chamber: is, that is one enough to respond? Or do I always wait for 2?

119
00:19:52.290 --> 00:20:01.120
senate chamber: And the the experimental design we need is to be able to selectively manipulate how quickly somebody's able to process each of those sources of information

120
00:20:02.200 --> 00:20:06.700
senate chamber: tends to be a little easier in simple visual signals where I can just reduce contrast.

121
00:20:06.900 --> 00:20:11.929
senate chamber: That's pretty good at slowing people down at detection. And I look at an interaction contrast

122
00:20:12.150 --> 00:20:16.519
senate chamber: of the speed up and slow down across each of those sources of information.

123
00:20:18.660 --> 00:20:30.899
senate chamber: So the whole setup is, you know, designed kind of like this, where you've got all the different combinations along with more information that you can get about the resource usage if you only have to deal with one source or the other.

124
00:20:31.680 --> 00:20:46.060
senate chamber: The 1st measure here is the mean interaction contrast. So this would be the same kind of thing you would see in a regression model. But it turns out this has actually pretty tight connections. To what type of architecture could have produced. What type of interaction contrast.

125
00:20:46.810 --> 00:20:55.979
senate chamber: So in particular, if I 1st focus on, let's say, let's focus on these 2 serial models.

126
00:20:56.120 --> 00:20:59.370
senate chamber: each of those predicts a mean interaction contrast of 0.

127
00:20:59.580 --> 00:21:07.210
senate chamber: Basically, you're speeding up and slowing down of each of those sources of information cancels out when you look at those contrasts, and you get a 0.

128
00:21:07.830 --> 00:21:20.309
senate chamber: On the other hand, if you take this exhaustive parallel process, so where you have to wait for the in parallel processing of both sources. That's going to give you a negative interaction. Contrast.

129
00:21:21.530 --> 00:21:26.340
senate chamber: This 1st terminating parallel process gives you a positive interaction contrast.

130
00:21:27.520 --> 00:21:31.660
senate chamber: and then the co-active also gives you a positive contrast.

131
00:21:32.370 --> 00:21:43.790
senate chamber: There's also so moving to the survivor. Interaction. Contrast the one I mentioned earlier rather than just taking the mean of your response time distributions. If we take the distribution

132
00:21:44.740 --> 00:21:51.099
senate chamber: or the survivor function was just this one minus the Cdf, so the probability that for each point in time.

133
00:21:51.330 --> 00:21:53.300
senate chamber: Response times are slower than that.

134
00:21:55.020 --> 00:22:07.239
senate chamber: We get a little more specificity and discriminating across these models. So in particular, while the mic for serial process is always 0. If you have this serial exhaustive, you get this 1st negative and then positive.

135
00:22:07.440 --> 00:22:12.060
senate chamber: whereas the the serial 1st terminating gives you an always flat line.

136
00:22:13.750 --> 00:22:18.839
senate chamber: Those of you familiar with math. An integrated survivor function is the mean for positive random variables.

137
00:22:19.550 --> 00:22:27.530
senate chamber: So you can also just look at area under the curve here to kind of connect to the mic. So area under the curve for the parallel exhaustive is negative.

138
00:22:28.020 --> 00:22:29.730
senate chamber: Hence the mic is negative.

139
00:22:30.525 --> 00:22:35.659
senate chamber: The coactive has the same kind of S-curve shape as the serial function.

140
00:22:36.480 --> 00:22:43.920
senate chamber: But if we recall back the mean interaction, contrast was positive. So we know this area under the curve, the integral should be more positive.

141
00:22:47.400 --> 00:22:49.930
senate chamber: I really should have put transition slides in. Sorry about that.

142
00:22:50.400 --> 00:22:58.690
senate chamber: Transitioning to our interpretation of chord assumptions of Act R. And again.

143
00:22:59.200 --> 00:23:08.630
senate chamber: you know, we kind of talk about this in the paper. This is one interpretation of core assumptions, and at this point we kind of wanted to transition to. If these are agreed upon core assumptions.

144
00:23:08.880 --> 00:23:15.770
senate chamber: what kind of conclusions can we make using the Sft tool as part of this global model analysis.

145
00:23:17.180 --> 00:23:40.420
senate chamber: and so 1st one, and this kind of goes back to some of the likelihood work that our team has done, that actar can be represented as a discrete time stochastic process, where the state space is a combination of discrete and continuous things. So state might be what's in what buffer. The continuous part might be activations or history of chunk usage.

146
00:23:41.246 --> 00:23:50.739
senate chamber: Declarative knowledge is representative chunks. So chunks consisting of spot value pairs where the activation is going to control both the speed and the likelihood of retrieval of a chunk.

147
00:23:53.240 --> 00:23:59.150
senate chamber: That the the processes are both modular and and in particular encapsulated.

148
00:24:00.190 --> 00:24:03.210
senate chamber: The production rules are executed once at a time.

149
00:24:03.470 --> 00:24:06.199
senate chamber: Modules can only process one request at a time.

150
00:24:06.700 --> 00:24:12.129
senate chamber: A buffer can contain at most one chunk, and each perceptual model encodes a stimulus into a single chunk.

151
00:24:15.522 --> 00:24:27.080
senate chamber: So the 1st theorem in our paper, then, was using those assumptions, and again the paper focused on perceptual things. So perceptual processing of multiple stimuli can only occur in serial.

152
00:24:27.360 --> 00:24:34.130
senate chamber: And this follows from the fact that modules can't be redundant. So we can't just put a new visual module in, or visual

153
00:24:35.110 --> 00:24:40.540
senate chamber: to, to deal with multiple things in the visual field. According to this assumption.

154
00:24:40.700 --> 00:24:47.980
senate chamber: because production rules are executed seriously and serially, also seriously.

155
00:24:49.390 --> 00:24:59.559
senate chamber: each stimulus is encoded into a single chunk, basically together. Those are saying, if I've got my dot up here and my dot down here. This 1, 1 of those gets put into

156
00:24:59.750 --> 00:25:03.000
senate chamber: the buffer, and the production wheel rules deal with it.

157
00:25:07.420 --> 00:25:21.300
senate chamber: so this has been tested. So the Townsend and Ozawa paper, where they 1st proposed this broad framework. Did that left and right visual field, one with system sectoral technology and found parallel processing across the board.

158
00:25:23.750 --> 00:25:38.240
senate chamber: Winger and Roten had a number of different tests where they were looking at visual stimuli, generally found actually a transition in training so early on more serial processing later with more expertise

159
00:25:38.690 --> 00:25:41.040
senate chamber: super capacity, parallel processing.

160
00:25:42.180 --> 00:25:57.469
senate chamber: So these, each of these experiments gives us evidence that we would use using the system factorial technology approach to reject that theorem. And you can't hold meaning that one of those assumptions, 4, 5, 6, or 8 are violated, based on

161
00:25:57.850 --> 00:26:00.249
senate chamber: information, perception, perceptual process.

162
00:26:01.330 --> 00:26:04.119
senate chamber: The next term was about cross-modal stimuli.

163
00:26:04.230 --> 00:26:09.409
senate chamber: And so the theorem here says that cross-modal stimuli cannot be co-active.

164
00:26:10.670 --> 00:26:24.779
senate chamber: So remember that's when it's sort of fused into a single source of information, and this comes from the encapsulation piece, the fact that each buffer can only hold one source of information, and then again, each each stimulus kind of has its own chunk.

165
00:26:32.410 --> 00:26:36.770
senate chamber: Alright. So in this setup couple. So

166
00:26:37.510 --> 00:26:43.860
senate chamber: Ian divine had a poster here where they're actually showing some cool activity just earlier in the week.

167
00:26:44.960 --> 00:26:52.299
senate chamber: Chengdao Young's lab has some results with this. There's also kind of the broader context. If you're familiar with

168
00:26:52.410 --> 00:26:57.290
senate chamber: audiovisual integration. There's a lot of work on showing multimodal enhancement.

169
00:26:58.267 --> 00:27:02.520
senate chamber: So again. These theorems.

170
00:27:03.330 --> 00:27:05.629
senate chamber: I guess there was another thing that dropped out.

171
00:27:06.220 --> 00:27:07.869
senate chamber: Oh, I see, I just didn't realize that

172
00:27:08.750 --> 00:27:12.250
senate chamber: the other one was the independent, unlimited capacity.

173
00:27:12.440 --> 00:27:21.100
senate chamber: so co-activity and independence in a moment, capacity across modality. And and this kind of collection of results indicates that these can't

174
00:27:22.550 --> 00:27:23.420
senate chamber: human perception.

175
00:27:25.340 --> 00:27:40.569
senate chamber: All right. So in conclusion, I think the the value of global model analysis using systems factorial technology is that we can use it to connect between some of these core structural assumptions and empirical testing, to evaluate

176
00:27:40.800 --> 00:27:43.859
senate chamber: much broader scale. That's not specific to

177
00:27:44.860 --> 00:27:56.380
senate chamber: one particular instantiation of an actar model or broadly architecture model and moves beyond. Just do these particular parameters satisfy a particular constraint.

178
00:27:59.180 --> 00:28:10.060
senate chamber: The serial constraints and the on intermodal processing and and sorry intermodal processing and parallel constraints on interim modal processing

179
00:28:10.930 --> 00:28:18.059
senate chamber: aren't supported empirically limitations. There's not a prediction about representation.

180
00:28:20.380 --> 00:28:25.720
senate chamber: although there is some work kind of looking to to build on that.

181
00:28:26.130 --> 00:28:32.550
senate chamber: Some of the sft models are not identifiable empirically. So you do get cases with when you lose encapsulation

182
00:28:32.710 --> 00:28:37.569
senate chamber: and have interactive systems where you start to see a breakdown in some of the nice, clean, deep

183
00:28:39.430 --> 00:28:41.840
senate chamber: distinction across the models.

184
00:28:42.100 --> 00:28:44.769
senate chamber: And then some predictions also may not be testable.

185
00:28:45.490 --> 00:28:47.060
senate chamber: Thank you very much. Sorry, for

186
00:28:55.370 --> 00:28:59.550
senate chamber: so all the panelists and the speaker should sit over there

187
00:29:02.750 --> 00:29:13.589
senate chamber: as a practical matter, we should all, every every place has their audio button, and you should use the audio button when you speak to be heard on zoom.

188
00:29:31.332 --> 00:29:35.989
senate chamber: All right. And David is virtual. Correct.

189
00:29:36.120 --> 00:29:40.962
senate chamber: Yeah, okay, so I'm gonna introduce the panel.

190
00:29:41.620 --> 00:29:48.940
senate chamber: we have joining us virtually. David Peevel. Who? David, can you? Well, there's nothing you can do. Say. Hi.

191
00:29:51.480 --> 00:29:52.660
David Peebles: Hi, everyone, everyone.

192
00:29:52.660 --> 00:29:59.079
senate chamber: There we go. Great yay! We also have Alex Huff.

193
00:29:59.360 --> 00:30:04.380
senate chamber: Rob Thompson, Christian Lebere, and Jan Javina, and

194
00:30:05.300 --> 00:30:10.469
senate chamber: I will do the 5 min timing, since Christian is otherwise occupied at the moment.

195
00:30:16.915 --> 00:30:18.980
senate chamber: And do we have a preferred order?

196
00:30:20.770 --> 00:30:29.469
senate chamber: We can. I had people listed in the order you wanted them to go. All those. Okay, Alex, you're up 1st perfect. Let me set the follow up, for everybody.

197
00:30:30.220 --> 00:30:33.600
senate chamber: Show you this alright testing.

198
00:30:35.140 --> 00:30:37.980
senate chamber: So I have a bunch of thoughts. But

199
00:30:38.854 --> 00:30:46.750
senate chamber: so I see this as like a potential model comparison tool as well. And there were some talks with Sft

200
00:30:47.360 --> 00:30:54.789
senate chamber: during the conference. That sort of got me thinking about some stuff. But I'm going to ask an easier question, and maybe all fun talk.

201
00:30:55.100 --> 00:30:57.070
senate chamber: So for the model comparison stuff.

202
00:30:57.190 --> 00:31:06.430
senate chamber: Would it be helpful to also either run an experiment or use Sft methods on a data set

203
00:31:06.750 --> 00:31:09.560
senate chamber: and also use on the model? Would that enable?

204
00:31:10.290 --> 00:31:18.220
senate chamber: So, for instance, like the perceptual separability and things like that, making sure that a lot of times we would do model fits. And

205
00:31:18.600 --> 00:31:20.520
senate chamber: I'm still even relatively.

206
00:31:22.060 --> 00:31:28.769
senate chamber: probably moderate experience for that color. So some of the model fitting we do, we'd like to do some more stuff on the

207
00:31:28.960 --> 00:31:35.019
senate chamber: worked in this area. So I was just thinking about. Not only if we do those model fits we have, but also fitting

208
00:31:37.010 --> 00:31:40.509
senate chamber: the order, and just more of the black box type stuff.

209
00:31:42.090 --> 00:31:51.219
senate chamber: Yeah. So so my my concept, I think of the workflow would be to start with a model that is a good representation of what you theoretically

210
00:31:52.320 --> 00:31:57.719
senate chamber: or or a collection of models that may be comparison. Set of the models.

211
00:31:58.354 --> 00:32:02.069
senate chamber: And then from there see if there are

212
00:32:02.571 --> 00:32:09.819
senate chamber: these global constraints that are testable, you know, either. I mean again, from my bias, the sft framework. But

213
00:32:10.110 --> 00:32:21.609
senate chamber: in these kind of global sense, testable distinctions across the models, in many cases there aren't going to be. It may come down to parametric differences, but I think the hope would be that

214
00:32:23.760 --> 00:32:31.659
senate chamber: that there are some distinctions, and if those distinctions map onto Sft, then the workflow would be to design the right experiment.

215
00:32:31.850 --> 00:32:36.200
senate chamber: to be able to then derive those measures and come to a conclusion about.

216
00:32:36.310 --> 00:32:42.279
senate chamber: does this particular model, indeed, perform better or better match the predictions?

217
00:32:43.040 --> 00:32:44.380
senate chamber: It's competitive models.

218
00:32:44.690 --> 00:32:47.069
senate chamber: So I just wrote model 2 data. So

219
00:32:48.270 --> 00:33:00.530
senate chamber: I see it as a feedback loop. I think I mean, I think we probably all do in this situation. It's, you know, there's always a crosstalk between the model or more broad level theoretical constructs and the data

220
00:33:02.980 --> 00:33:05.979
senate chamber: think you have 2 and a half minutes.

221
00:33:07.800 --> 00:33:08.880
senate chamber: So one

222
00:33:09.720 --> 00:33:16.230
senate chamber: sorry. So one thing I was also thinking about from some watching, some sft, and I'm a bit ignorant on on the whole.

223
00:33:16.490 --> 00:33:25.309
senate chamber: just make that clear. So I know it's mostly based on perceptual stuff so like an area that I'm really interested in. I get to talk about is like influence and

224
00:33:27.010 --> 00:33:30.369
senate chamber: kind of figuring out how to disentangle some of the effects.

225
00:33:30.540 --> 00:33:36.750
senate chamber: So that struck me there was a talk with, I mean. Murray gave a talk, looking at implicit attitudes.

226
00:33:36.940 --> 00:33:49.080
senate chamber: So that struck me as interesting because a lot of times. You look at the perceptual. Is there the possibility of extending to things like multiple queue decision making where we're talking about qualities not necessarily separable

227
00:33:49.810 --> 00:33:50.800
senate chamber: modalities.

228
00:33:52.300 --> 00:33:56.160
senate chamber: Yes, but so I I mean.

229
00:33:57.000 --> 00:34:04.479
senate chamber: I think, driven in part by my background in sort of areas of comfort and expertise. Mostly I've worked in the perceptual domain.

230
00:34:05.200 --> 00:34:12.099
senate chamber: And so my, my quick thought responses. I was like, Oh, well, we've done, we've done, but it's usually mapped into a perceptual task.

231
00:34:12.420 --> 00:34:15.800
senate chamber: However, the theory is more about

232
00:34:16.778 --> 00:34:24.810
senate chamber: essentially, how quickly people are using information. So there's nothing theoretically preventing one from moving to

233
00:34:24.969 --> 00:34:32.559
senate chamber: more complex types of information. Mario Fifich has some great work looking at

234
00:34:32.780 --> 00:34:41.150
senate chamber: memory, retrieval types of things. Character, recognition, I think, was the early stuff, or- or, you know.

235
00:34:41.440 --> 00:34:43.960
senate chamber: match to sample kind of characters.

236
00:34:44.980 --> 00:34:52.010
senate chamber: there's there's certainly Murray's stuff that he talked about this week. There was some work at 1 point, looking at.

237
00:34:53.739 --> 00:34:55.580
senate chamber: Remember the exact details.

238
00:34:56.489 --> 00:35:03.409
senate chamber: essentially, how how comfortable people felt with an argument based on different parts of the argument.

239
00:35:03.860 --> 00:35:09.289
senate chamber: So I think if as long as you can map onto that kind of speeding up, speeding up and slowing down part

240
00:35:09.510 --> 00:35:18.359
senate chamber: and recover strong enough signal and empirical data to get that survival interaction. Contrast, then, in principle, it can be used.

241
00:35:19.520 --> 00:35:25.579
senate chamber: So, yeah, it does sound like it can be because I'm thinking about if you think about some traditional strategy

242
00:35:25.690 --> 00:35:33.429
senate chamber: decision making strategies like, take the best. So that is faster, because less information is considered. So that's kind of the things I was thinking about. So it sounds like

243
00:35:33.640 --> 00:35:39.659
senate chamber: it's possible it might take some effort. But I was just kind of thinking about if it was possible to use the net.

244
00:35:40.230 --> 00:35:47.757
senate chamber: Yeah. And we've done some stuff with heuristic decision making. But it it seems like kind of a bigger example, because it's all mapped on the perceptual decision making. So like

245
00:35:48.460 --> 00:35:52.950
senate chamber: the Beatles task, we're comparing a bunch of those. Remember that one. So we've got some of that.

246
00:35:55.320 --> 00:35:56.340
senate chamber: All right.

247
00:35:57.270 --> 00:35:58.060
senate chamber: Move on.

248
00:35:59.940 --> 00:36:04.940
senate chamber: It's you slides

249
00:36:18.110 --> 00:36:20.009
senate chamber: first.st Thank you.

250
00:36:21.140 --> 00:36:26.399
senate chamber: There's great idea to discuss papers that are already published.

251
00:36:31.340 --> 00:36:34.489
senate chamber: Also, thanks to Joe for coming here.

252
00:36:36.000 --> 00:36:43.000
senate chamber: Yeah, the way the speaker, SSD, and few things in common

253
00:36:46.620 --> 00:36:51.040
senate chamber: to a lot of capacity rounding model even bigger.

254
00:36:54.420 --> 00:36:57.680
senate chamber: Some caveats disclaimer

255
00:36:57.910 --> 00:37:03.810
senate chamber: my critique of this paper stems from my long term interest in higher order and integrated cognition.

256
00:37:04.840 --> 00:37:09.420
senate chamber: not just perception, but I should say a little about perception.

257
00:37:11.330 --> 00:37:14.629
senate chamber: In my relatively recent focus on environmental.

258
00:37:15.380 --> 00:37:17.590
senate chamber: it is how the environment shapes.

259
00:37:21.300 --> 00:37:32.055
senate chamber: So speaking about assumptions, I think the distinction between core and auxiliary assumptions is important, and

260
00:37:32.910 --> 00:37:34.390
senate chamber: join his colleagues.

261
00:37:34.760 --> 00:37:43.140
senate chamber: Great job, we can get a distinction, but I think actor has many simplifying assumptions.

262
00:37:43.380 --> 00:37:46.550
senate chamber: These are not necessarily for an auxiliary.

263
00:37:46.700 --> 00:38:00.140
senate chamber: Both of them can be simplified, and that was necessary for translating the theory into this is

264
00:38:00.830 --> 00:38:04.350
senate chamber: important for this approach to modeling

265
00:38:05.620 --> 00:38:09.830
senate chamber: actori's position at a level of abstraction

266
00:38:10.350 --> 00:38:19.420
senate chamber: that allows you to study functional cognition and behavior that is doing particular tasks so end to end behavior

267
00:38:19.640 --> 00:38:23.470
senate chamber: from starting the task to finishing it and achieving the outcome. So

268
00:38:25.390 --> 00:38:31.450
senate chamber: for that reason, sometimes it necessarily abstracts out a lot of details of the function.

269
00:38:32.120 --> 00:38:34.539
senate chamber: and it summarizes brain activity with mathematical.

270
00:38:35.490 --> 00:38:37.160
senate chamber: So this is simplification.

271
00:38:37.980 --> 00:38:39.830
senate chamber: We know that our models are

272
00:38:43.130 --> 00:38:49.970
senate chamber: core assumption of Aktar theory that processing within modules is massively parallel.

273
00:38:51.110 --> 00:38:57.780
senate chamber: There are few bottlenecks and bottlenecks are important

274
00:38:58.000 --> 00:39:04.560
senate chamber: for a functional architecture. They're not just ways to slow things down or just limitations.

275
00:39:05.260 --> 00:39:08.779
senate chamber: Bottleneck is is a mechanism of choice.

276
00:39:09.060 --> 00:39:15.799
senate chamber: So it helps people, helps cognition separate, relevant from irrelevant.

277
00:39:16.790 --> 00:39:20.260
senate chamber: So the core assumption is really that there are bottlenecks.

278
00:39:21.330 --> 00:39:29.360
senate chamber: Fact that the bottleneck is one chunk, or one rule is a simplifying assumption

279
00:39:29.840 --> 00:39:32.029
senate chamber: principle. It could be 2 or 3.

280
00:39:32.920 --> 00:39:36.470
senate chamber: But court assumption is that there are bottlenecks.

281
00:39:37.720 --> 00:39:41.110
senate chamber: and we know, for example, soar relaxes

282
00:39:41.490 --> 00:39:48.029
senate chamber: the bottlenecks, but they still have bottlenecks, because bottlenecks are important achieving functionality.

283
00:39:51.140 --> 00:39:56.870
senate chamber: The paper focuses on latencies, that is durations.

284
00:39:57.050 --> 00:40:03.529
senate chamber: And that is important because it indicates that serial or parallel processing

285
00:40:05.170 --> 00:40:12.920
senate chamber: But Athar is also concerned with the timing of actions, that is, when actions happen in a sequence

286
00:40:13.750 --> 00:40:17.719
senate chamber: that is, food behavior and the strategy

287
00:40:17.920 --> 00:40:20.630
senate chamber: for a variety of tasks and environments.

288
00:40:23.810 --> 00:40:28.280
senate chamber: So be a little bit faster.

289
00:40:28.490 --> 00:40:37.759
senate chamber: Duration and timing of actions are determined not only by whether processing is serial or parallel, but also by the requirements and constraints of the task. Environment.

290
00:40:39.280 --> 00:40:43.529
senate chamber: Cognition is not just produced by the interaction between perception, memory and action.

291
00:40:43.700 --> 00:40:47.740
senate chamber: but also by the interaction between all of these and the task environment.

292
00:40:47.890 --> 00:40:56.050
senate chamber: So the dynamics of the task environment might impose either serial or parallel processing. If I have to do parallel processing.

293
00:40:56.250 --> 00:40:58.629
senate chamber: I find a way to do it. Commission finds a way.

294
00:41:02.140 --> 00:41:10.979
senate chamber: and lastly, and very shortly, here the paper takes this approach of barbarian falsifiability

295
00:41:11.210 --> 00:41:24.079
senate chamber: important, but some of my topic. And I suggest also, considering what's going their approach is suggesting that

296
00:41:24.480 --> 00:41:27.099
senate chamber: theories can be defended and amended.

297
00:41:27.320 --> 00:41:29.309
senate chamber: If they have a good track record

298
00:41:29.450 --> 00:41:34.910
senate chamber: of making successful or near miss predictions, no prior probabilities.

299
00:41:35.040 --> 00:41:38.720
senate chamber: These are predictions that that surprise us.

300
00:41:47.720 --> 00:41:48.640
senate chamber: Christian.

301
00:41:51.780 --> 00:41:53.270
senate chamber: Some people at Zoom are having

302
00:41:53.500 --> 00:41:55.370
senate chamber: here in the panel. So it's okay.

303
00:41:55.640 --> 00:41:58.729
senate chamber: Make sure you're close enough to the mic. There, how's this?

304
00:41:59.750 --> 00:42:13.059
senate chamber: So 1st of all, and I think that builds on what Jan was just saying. I love this approach of trying to falsify theories, and in particular, I think a lot of us were

305
00:42:13.110 --> 00:42:34.640
senate chamber: affected by Roberts and passion. How persuasive is a good fit model! The fact! It's not just what you can fit. It's what you cannot fit. And in those days, anticipating the next session on Centaur. If you have trillion parameters, you can fit everything, and it's opposite. So I think that this is particularly important these days.

306
00:42:34.910 --> 00:42:59.720
senate chamber: One thing that's always been hard about falsifying or verifying cognitive architectures is how flexible they are. So I was at some marshake session yesterday, where basically the kind of decisions flow that Joe described was referred to as a cognitive architecture. We have more wiggle room than that. So that's always sort of a difficult thing

307
00:42:59.720 --> 00:43:05.270
senate chamber: to PIN down. So, having some formal way of going about, that is great.

308
00:43:05.270 --> 00:43:12.320
senate chamber: Some specific aspects of it are really quite interesting to me. So, for example, I wondered whether your capacity

309
00:43:12.320 --> 00:43:30.510
senate chamber: is related to, for example, the W. Parameter in actor that we've associated with working memory could also be associated with things like the temperature and blending. So it would be really great to try and draw links between those capacity parameters and architecture parameters.

310
00:43:30.690 --> 00:43:58.120
senate chamber: And, as I said, we have a lot of wiggle room in the in the architecture in particular, as Jan was also alluded. This is what I call the gray zone between the theory and the implemented system. And we make a lot of decisions for sort of practical reasons. And sometimes they're not even decisions. They just happen. So so in particular, here's 3

311
00:43:58.490 --> 00:44:11.590
senate chamber: shortcomings that I think are not fundamental shortcomings of the architecture that are current shortcomings of the implemented system that your analysis suggested to me. The 1st one is that there is no learning in perception

312
00:44:11.770 --> 00:44:23.700
senate chamber: which is just crazy, right? I mean, we. We know that we, our perception in particular, that addresses, for example, that issue of processing a single stimuli versus multiple stimuli

313
00:44:23.710 --> 00:44:43.539
senate chamber: chunks. You know, Simon told us that we learn patterns right. And then multiple stimuli can become one pattern. So the fact that we don't have learning perception is a clear shortcoming of the current system. It's not in the theory. And indeed, John had a rational analysis of perceptual categorization. So we could, we could, we should develop that right.

314
00:44:43.640 --> 00:45:08.359
senate chamber: Another issue that, I think, is related is the fact that we treat perception, particular visual perception, but other ones as well, and memory as completely separate modules. And you know, I think, from a neuroscience perspective that's somewhat dubious. But and obviously memory learns right? So that would be potentially a way of trying to get learning into perception.

315
00:45:08.360 --> 00:45:14.159
senate chamber: But again, it's not particularly architectural assumption other than the general framework of modularity

316
00:45:14.420 --> 00:45:37.500
senate chamber: and the final one about the inability of doing cross-modal representation. Again, that's not the fact that each module has their own buffer in their own chunk doesn't see anything about what happens downstream. And, in fact, Rob developed an episodic memory module that did exactly that that provided essentially cross-module integration around the buffer.

317
00:45:37.500 --> 00:45:47.939
senate chamber: So again, it's sort of like the difference between the system as it is now, and it is implemented versus what it could be. So I think from from this perspective, this is an impetus for us to

318
00:45:48.210 --> 00:45:52.939
senate chamber: to to push and to do better, and to to keep evolving the architecture

319
00:45:59.250 --> 00:46:01.080
senate chamber: alright.

320
00:46:01.340 --> 00:46:03.020
senate chamber: So next up David.

321
00:46:04.180 --> 00:46:05.919
David Peebles: I am here. Can everybody hear me?

322
00:46:06.880 --> 00:46:07.730
senate chamber: Yes, we can hear you.

323
00:46:07.730 --> 00:46:09.170
David Peebles: Yeah, okay, thank you.

324
00:46:11.370 --> 00:46:14.329
David Peebles: Thank you. Yeah. I'd just like to. I mean.

325
00:46:15.590 --> 00:46:25.807
David Peebles: repeat what others have said, that I think this is a fantastic piece of work, and it's a really important development in thinking about constraints at the architectural level.

326
00:46:26.970 --> 00:46:34.119
David Peebles: I just so I've got a couple of questions really. I just wondered what the authors think about what the

327
00:46:34.330 --> 00:46:37.539
David Peebles: the scope of this form of analysis is.

328
00:46:38.020 --> 00:46:38.690
David Peebles: M.

329
00:46:39.100 --> 00:46:41.920
David Peebles: But in terms of different aspects of the architecture.

330
00:46:42.150 --> 00:46:43.690
senate chamber: But also to.

331
00:46:43.810 --> 00:46:46.140
David Peebles: More complex tasks.

332
00:46:46.530 --> 00:46:53.639
David Peebles: So it strikes me that the the architecture might be for sort of minimal tasks. It constrains

333
00:46:53.940 --> 00:46:58.330
David Peebles: quite significantly the possibilities. But for more complex tasks

334
00:46:59.860 --> 00:47:04.290
David Peebles: you've got strategy and adaptive behaviour that becomes more relevant.

335
00:47:04.520 --> 00:47:07.109
David Peebles: Does that limit the predictive power of

336
00:47:07.560 --> 00:47:13.180
David Peebles: architectural constraints that are identified in this minimal tasks?

337
00:47:13.710 --> 00:47:15.569
David Peebles: Can I ask that question first? st

338
00:47:15.570 --> 00:47:18.779
senate chamber: Sure. Yeah, I think. What's it?

339
00:47:18.900 --> 00:47:22.566
senate chamber: Theoretically? No. I think there's a lot of opportunity, maybe through

340
00:47:23.480 --> 00:47:28.799
senate chamber: through analysis of the model, to determine in much more complex tasks.

341
00:47:29.140 --> 00:47:40.289
senate chamber: when you would expect to see parallel or serial information processing, when you would expect to see limited super capacity or cases of co-active processing.

342
00:47:41.160 --> 00:47:47.879
senate chamber: I think a potential limiting factor, though, would be the empirical test. So

343
00:47:48.090 --> 00:47:56.460
senate chamber: these the data sets I was talking about here were, you know hundreds, if not thousands, of trials, on fairly simple stimuli.

344
00:47:56.990 --> 00:48:05.190
senate chamber: having a person in a more complex task go through that much, that much experimentation could be fairly limited.

345
00:48:05.860 --> 00:48:12.690
senate chamber: There's there's work we've done on more hierarchical modeling. You know how much you can assume similarity across people, and then

346
00:48:13.360 --> 00:48:20.159
senate chamber: build a more general model of of people rather than a person might partially address that.

347
00:48:21.520 --> 00:48:26.529
senate chamber: Yeah, I guess the short answer is, theoretically, no problem empirically could be a problem.

348
00:48:28.150 --> 00:48:38.029
David Peebles: Thank you. My second question really was, how might these architectural constraints be integrated with constraints from

349
00:48:38.550 --> 00:48:42.612
David Peebles: the computational and implementation levels? So

350
00:48:44.430 --> 00:48:50.249
David Peebles: I guess that the ideal one where you get sort of convergence were sft and fmri

351
00:48:50.410 --> 00:48:56.450
David Peebles: and rational analysis, all suggest serial processing.

352
00:48:56.660 --> 00:49:04.060
David Peebles: And then you've you know, obviously, you've got areas where there may well be conflict, particularly where, say, for example, your neuroscience

353
00:49:05.010 --> 00:49:15.710
David Peebles: shows parallel processing where behavioral and maybe rational analysis might suggest sequential.

354
00:49:18.900 --> 00:49:26.779
David Peebles: I know it's a general question, really. You know, how do people think that

355
00:49:27.420 --> 00:49:29.549
David Peebles: these sorts of analyses might

356
00:49:31.170 --> 00:49:39.459
David Peebles: produce that multi-level set of constraints on the architecture, which I think is probably what we're, you know, all looking for.

357
00:49:42.040 --> 00:49:47.779
senate chamber: Yeah, I mean, I think it's it's a really interesting question. And probably the fundamental question of how we integrate across levels

358
00:49:48.190 --> 00:49:54.870
senate chamber: from the sft. Perspective it it. What you're testing with, you know, with those route interaction contrasts

359
00:49:55.030 --> 00:49:58.409
senate chamber: is the level at which you're manipulate your your

360
00:49:59.990 --> 00:50:02.710
senate chamber: stimulus manipulation is influencing the system.

361
00:50:03.990 --> 00:50:15.220
senate chamber: So if I'm doing things perceptually that are are changing things at the retinal level, then I'm asking a question of, are things happening in parallel or not at the retinal level. If I'm asking a question about

362
00:50:16.030 --> 00:50:21.499
senate chamber: spatial comparisons, then, even if things might be

363
00:50:21.610 --> 00:50:29.010
senate chamber: happening in serial early in the visual system like, I'm making multiple fixations on a scene. It could still be that the

364
00:50:29.200 --> 00:50:32.670
senate chamber: the location, information, or comparisons happening in parallel.

365
00:50:33.330 --> 00:50:36.830
senate chamber: And so for the sft part, it really comes down to

366
00:50:36.970 --> 00:50:40.409
senate chamber: where your your influence manipulation is hitting the system.

367
00:50:41.410 --> 00:50:42.499
David Peebles: Okay, thank you.

368
00:50:43.920 --> 00:50:47.995
David Peebles: Yeah, that's it. Really. I think it's, you know, it's a really exciting development. I think.

369
00:50:48.560 --> 00:50:55.703
David Peebles: having that those architectural level constraints and then trying to see how they integrate with other levels.

370
00:50:56.240 --> 00:50:58.490
David Peebles: you know, it provides a really.

371
00:50:59.130 --> 00:51:02.550
David Peebles: you know, promises really important development in the theory.

372
00:51:03.610 --> 00:51:05.909
David Peebles: I think that's that's it. That's my comments.

373
00:51:11.235 --> 00:51:12.600
senate chamber: Alright. So, Rob.

374
00:51:15.880 --> 00:51:38.210
senate chamber: thank you. One of the things that was interesting across what everybody was talking about is kind of this focus on what are some of the core constraints? And how could we use Sfg to potentially evaluate some of them? You know, actor has been great for the having the modularity to be able to sub in various alternatives.

375
00:51:38.510 --> 00:51:53.999
senate chamber: I can't remember who did it, but until it was about a decade ago there was a maybe a little bit more than that, there was a pre-attentive and attentive vision module that came out, that that allowed for perceptual priming some multi, something a little bit.

376
00:51:54.220 --> 00:52:02.540
senate chamber: you know, one level further down on the perceptual land, and that would be very interesting to be evaluated to see. Can we challenge some of these assumptions as we go through.

377
00:52:05.650 --> 00:52:12.719
senate chamber: One of the interesting things like I'd be, I'd like to

378
00:52:13.170 --> 00:52:36.189
senate chamber: investigate as we look at. Look at Sft is also in the production system. Because, you know, we're talking. We're talking broad modules, production being one of them. And in that case strategy selection does come with all the bindings and the various serial and parallel things that would let us kind of evaluate the kind of the multiple ways to skin a cat.

379
00:52:36.734 --> 00:52:54.085
senate chamber: You know, last year Christian and I were studying some influence. We're studying some studying influence dynamics looking at conformity. And in the process of that I was doing a completely representation based blending account. And Christian was evaluating

380
00:52:55.260 --> 00:52:56.100
senate chamber: to

381
00:52:56.560 --> 00:53:15.769
senate chamber: modules with similarity, learning and structure mapping. And it's essentially getting to similar outcomes. And so in the end. If we use something like Sft to be like, what unique assumptions can we make that would help us tease, you know.

382
00:53:16.320 --> 00:53:18.530
senate chamber: future development on either on either end.

383
00:53:21.900 --> 00:53:30.669
senate chamber: Yeah, I mean, I don't. I don't have a specific comment that sounds really exciting to to discuss further. I think that would be really interesting. Is there a way to to integrate those 2?

384
00:53:35.740 --> 00:53:50.009
senate chamber: Yeah, 2 and a half minutes, Rob? The 5 min is a maximum. He looked at me like, how much time do I have left? So I was trying to answer. There are also 10 comments in the chat.

385
00:53:50.210 --> 00:53:51.390
senate chamber: What about

386
00:53:53.350 --> 00:54:19.360
senate chamber: all right? Well, Rob is done so. I want to open it to the floor. But I'm going to use my limited amount of authority as the session chair to start by, even though I'm not a panelist reacting in that. I think these kinds of questions and discussions have a history of being a lever for change in actar. I think lots of times in the past

387
00:54:19.430 --> 00:54:36.020
senate chamber: we have run into something where someone says, Look, people clearly can do this. Actar can't do this. We have to fix, change, alter whatever. And as the architect of one of those things, so when I arrived at Cmu, Actar couldn't dual task.

388
00:54:36.260 --> 00:55:02.530
senate chamber: not really. And one of the things that I did was, you know, rebuilt part of Act R, so that we could sort of try to dual task, and what we did 1st was. Well, let's see if we can model the simplest dual task experiments we can find, which is the Prp which you do not want to get into, which is a deep rabbit hole of very strange experiments. Right? But I really appreciate this because I do think that this is a motivator for us

389
00:55:02.580 --> 00:55:11.950
senate chamber: to think hard about what we can and can't. Do. You know, in the current implementation, in ways that are consistent with what we have, but also informed by.

390
00:55:12.080 --> 00:55:16.349
senate chamber: you know more, not more new methods, methods of new to us.

391
00:55:16.780 --> 00:55:18.480
senate chamber: Alright, someone else.

392
00:55:22.580 --> 00:55:33.299
senate chamber: So to quote a comment, actually, somebody comment quoting somebody from yesterday.

393
00:55:34.000 --> 00:55:49.230
senate chamber: all theory. We know all theories are wrong. Some theories are useful, and that's certainly, I think, been behind at R. And I think what we see is that Sft is useful, too, to go back to. I guess

394
00:55:49.390 --> 00:56:04.090
senate chamber: Mike's influence and and the history of Act R. Act R. Became a lot more useful when essentially Mike provided us with the perceptual motor interface to act R,

395
00:56:04.500 --> 00:56:33.300
senate chamber: and it's been, and I think, as Christian and others have remarked, we've always known that our perceptual processing was kind of thin and inadequate, and this motivates the development of a richer approach to perceptual processing. Then that's good. It's been something of a question to me, why more hasn't happened

396
00:56:33.440 --> 00:56:40.550
senate chamber: on that front over all this time? Because I think we've been aware pretty well that there are a lot of things that

397
00:56:41.550 --> 00:56:47.140
senate chamber: learning being, I guess, the most prominent that just weren't weren't being properly represented.

398
00:56:50.790 --> 00:56:57.779
senate chamber: We may want to alternate between taking questions from your comments from the room and online. Let you guys manage that.

399
00:57:02.360 --> 00:57:06.979
senate chamber: Nobody online has a hand up that I can see. So someone from the room wanted to jump in.

400
00:57:08.300 --> 00:57:09.000
senate chamber: No

401
00:57:10.810 --> 00:57:22.149
senate chamber: see if mine works. Yeah, it does. Hi, everybody. I have a question about system tutorial technology. But let me 1st motivate the question.

402
00:57:22.430 --> 00:57:28.970
senate chamber: So I remember, many years ago I was very inspired by Karl Popper and philosophers of science.

403
00:57:29.510 --> 00:57:32.059
senate chamber: using logic to do science.

404
00:57:33.390 --> 00:57:38.959
senate chamber: Now I'm a little less enthusiastic because logic is binary.

405
00:57:39.460 --> 00:57:48.039
senate chamber: and and we are dealing with these hazy, vague, hard to study objects with limited empirical access to

406
00:57:48.150 --> 00:57:54.190
senate chamber: and and so reiterate the point. Models are approximations

407
00:57:55.160 --> 00:58:06.740
senate chamber: to keep it simple. Imagine a function which isn't quite linear, but it is linear enough so that the linear model is doing a good job like a linear regression accounts for

408
00:58:07.080 --> 00:58:09.480
senate chamber: 85% of the variance.

409
00:58:09.700 --> 00:58:13.640
senate chamber: Now, from a logical point of view, it is not linear.

410
00:58:13.780 --> 00:58:24.639
senate chamber: If you have enough statistical power and proper tools, you can prove that it is stagnant, statistically, significantly, nonlinear.

411
00:58:25.300 --> 00:58:27.729
senate chamber: And so it isn't linear. But

412
00:58:28.260 --> 00:58:34.709
senate chamber: again, a linear regression accounts for 90 85% of the variance. And so

413
00:58:34.870 --> 00:58:47.449
senate chamber: here's the question. Finally, even if something is parallel or it has a parallel component, does this sft technology allow to somehow measure.

414
00:58:48.570 --> 00:58:56.889
senate chamber: though, is there some analog to like? It's 85% serial and 15% parallel, something like that.

415
00:58:58.390 --> 00:59:11.679
senate chamber: Yes, sir, there is that possibility. So the theory as I've presented it so far, is much more about the sort of falsification, you know. Can we reject full classes of models

416
00:59:12.760 --> 00:59:20.750
senate chamber: and do so in a way, I mean, whenever we're dealing with data, there's statistics involved. But to do so in a way that's not just.

417
00:59:20.860 --> 00:59:24.069
senate chamber: hey, if we make a normal distribution assumption. Things are bad.

418
00:59:27.720 --> 00:59:42.590
senate chamber: so the the short answer to the question is, we can also look at mixture models. So perhaps there are situations where, on some trials, people are relying more on serial information processing and on others parallel information processing.

419
00:59:43.840 --> 00:59:45.205
senate chamber: We see that

420
00:59:46.900 --> 01:00:01.699
senate chamber: remember the exact context of the experiment. But in parametric model fits that has shown up sometimes in decision-making experiments, people might consider, you know, both amount and probability on some trials, sequentially and other maybe do more parallel processing.

421
01:00:04.330 --> 01:00:07.285
senate chamber: That then, I think, becomes

422
01:00:09.080 --> 01:00:21.449
senate chamber: maybe indistinguishable from uncertainty about how much of a parallel or how much of a serial thing is happening always without being able to get at single trial level analysis

423
01:00:22.180 --> 01:00:45.549
senate chamber: and maybe comment a little more broadly on the philosophy of science piece, because it's come up as well. I think so. I absolutely agree that we can't have a science that's just about falsifying things, and we always are bringing more assumptions to the table that it's never quite clear if it's our ancillary assumptions or simplifying assumptions that lead to the falsification.

424
01:00:47.040 --> 01:00:52.959
senate chamber: And I think it. You know this idea of hoping that we're walking towards truth, or

425
01:00:53.230 --> 01:01:00.780
senate chamber: continuing to remove things from from the field of consideration, that we are sure is wrong.

426
01:01:00.970 --> 01:01:16.750
senate chamber: and I think where I would position. This work is more if we can be confident that things are wrong, that not continuing to attend to them as we move forward in our science allows us to focus on things that may still be right or may still be closer to right.

427
01:01:23.129 --> 01:01:24.949
senate chamber: Yeah, Andrea.

428
01:01:26.480 --> 01:01:29.209
Andrea Stocco: Hey, everybody? Thank you. So

429
01:01:29.880 --> 01:01:38.859
Andrea Stocco: first, st I want to really thank Joe and the co-autos, because this is like a fantastic paper, and also honestly, a fantastic presentation of a difficult paper.

430
01:01:40.310 --> 01:01:48.730
Andrea Stocco: I really like the discussion, and maybe I said something redundant here. But one of the things that for me is difficult to take away is.

431
01:01:48.990 --> 01:02:02.200
Andrea Stocco: where is the problem? So perception is a great example. I think everybody knows the perception is a great approximation, but also a big approximation to what is going on when people perceive things.

432
01:02:02.590 --> 01:02:09.409
Andrea Stocco: and it is my impression that also, also from your slides, that experiments in perception

433
01:02:09.990 --> 01:02:15.549
Andrea Stocco: are highly sensitive to the stimuli. Right? Is it serial? Can you proceed to things at once? Well.

434
01:02:15.670 --> 01:02:18.660
Andrea Stocco: probably you can proceed to color and to gabor patches.

435
01:02:19.050 --> 01:02:25.419
Andrea Stocco: not to words, probably, and definitely, not to objects that are complex, like at the level of semantics.

436
01:02:26.810 --> 01:02:41.930
Andrea Stocco: the point that I'm trying to make is like, okay. Now that we know that at a certain point Akta struggles with this, and this is probably an important limitation. Where is the limitation coming from? Is it the way that the module

437
01:02:42.120 --> 01:02:47.340
Andrea Stocco: works? Is it the way the perception, maybe, should be broken into different modules?

438
01:02:47.620 --> 01:03:01.010
Andrea Stocco: Or is it the way in which we represent information. And maybe at a certain point, chunks are really, we're pushing the boundaries here and there is a point when you're looking again like a luminous contrast where chunks are just not the best way to represent information at all.

439
01:03:01.140 --> 01:03:12.429
Andrea Stocco: And I was wondering whether you have thought about like ways in which this technique could be used to pinpoint exactly the root cause of the problem in a computational framework.

440
01:03:17.310 --> 01:03:20.909
senate chamber: That's a really big question. So

441
01:03:21.020 --> 01:03:31.529
senate chamber: we've certainly put thought into it. It can be difficult, you know, the the way we'd set up the theorems. It was really we needed to rely on the collection of assumptions

442
01:03:31.710 --> 01:03:34.020
senate chamber: for that theorem to apply broadly.

443
01:03:34.660 --> 01:03:41.529
senate chamber: I think if there are situations where some of those assumptions are less likely to hold true.

444
01:03:42.580 --> 01:03:47.909
senate chamber: So and do we really want to suggest strict encapsulation of

445
01:03:48.563 --> 01:04:03.109
senate chamber: audio visual information given? What we know about some of the cross modal integration things. You know, perhaps starting with, if if that's the assumption we drop for cross model information, how well, then, does the

446
01:04:03.720 --> 01:04:10.360
senate chamber: does the architecture, the actor architecture, reflect the data that we've collected in that domain?

447
01:04:14.170 --> 01:04:18.819
senate chamber: At least with what we have so far, I'm not sure that there is a way to

448
01:04:19.240 --> 01:04:26.839
senate chamber: to know exactly which assumption certainly across the the 3 theorems there were some shared assumptions.

449
01:04:27.000 --> 01:04:31.649
senate chamber: so if we dropped one assumption that's shared across all 3.

450
01:04:31.960 --> 01:04:37.710
senate chamber: Maybe it. It solves all those problems. But that also may not be the right thing for the theory. So I don't know

451
01:04:37.880 --> 01:04:41.190
senate chamber: in front of me, but I can imagine some of those shared theorems

452
01:04:41.440 --> 01:04:45.709
senate chamber: wouldn't be as comfortable to drop, because that seems to step further away from the theory.

453
01:04:49.860 --> 01:04:50.740
Andrea Stocco: Thank you.

454
01:05:00.320 --> 01:05:07.399
senate chamber: I detect quiescence already. That's too old. That's way over.

455
01:05:09.610 --> 01:05:26.009
senate chamber: Hello, Mark, or I'm I'm outside of the actor community, and you'll see me talk soon, and you can get to know me a little bit, then. So this question is somewhat uninformed. But to me the and I don't know the systems.

456
01:05:26.910 --> 01:05:31.259
senate chamber: electoral technology. Well, I didn't read the paper. I apologize, but

457
01:05:31.510 --> 01:05:56.880
senate chamber: Christian and IA couple of years ago, were working with some mathematicians who use this very particular kind of graph systems to do proofs on graphs about the dynamics of systems. It's called graph dynamical systems. It's very particular for certain kinds of applications. We were interested in understanding some of the bounds on actor models of attitudes that we were trying to develop. And from that work

458
01:05:56.960 --> 01:06:02.620
senate chamber: the basic idea was, take Actar, put it into some kind of representation that the mathematicians can use to do proofs on.

459
01:06:03.010 --> 01:06:20.210
senate chamber: and you can say that kind of general procedure is applicable all across the board for act R, and it would be a separate. You use that procedure in the abstract, but it seems like it's very general, it can be applied across all. You can come up with all kinds of things about Act R. You want to prove. I'm sure John has

460
01:06:20.530 --> 01:06:25.419
senate chamber: some things in mind from from. So the rational analysis for the

461
01:06:25.610 --> 01:06:34.919
senate chamber: the 1989 book or 1990 book, there was a lot of work on the Bayesians. It wasn't proof, necessarily, but it was pointing in that direction to some degree.

462
01:06:35.020 --> 01:06:36.269
senate chamber: And you know.

463
01:06:36.310 --> 01:06:49.930
senate chamber: formal systems and formal methods in computer science chip design all that stuff. It's all in the same approach. And the problem with the formalization is a serious problem, because it does deviate sometimes from the system that you're actually looking at.

464
01:06:49.950 --> 01:07:13.130
senate chamber: So to this community here I'm not aware of. But where are the other proofs? It doesn't necessarily come from the tasks that are looking perceptual, and then going from perceptual and going into more complicated tasks and tasks with more flexibility and tasks where the environment is feeding back. But it seems like there's if you wanted to. There's all kinds of ways you could enter this area. And I was wondering, where is that work? And

465
01:07:14.680 --> 01:07:25.470
senate chamber: I just wanted to remark that this technology is just one approach in that general scheme, and I think if you're going to go that way, you should broaden the net. And I'll stop with that. Thank you.

466
01:07:27.660 --> 01:07:51.239
senate chamber: I think that's a really good point to mention another example. So a few years ago, we developed a Covid model and specifically a model of behavior. Now the technology for epidemiological models is by and large ordinary differential equations. And actually we did quite a bit of progress, especially Konstantinos Mitzopulos.

467
01:07:51.240 --> 01:08:03.530
senate chamber: On expressing the actar model in terms of ordinary differential equations that can be done either computationally.

468
01:08:03.580 --> 01:08:29.789
senate chamber: you can sort of capture the dynamics and fit an equation to that. That sort of instance-based learning type approach. But it's also, I think, you know, it seems like, for example, the memory system in actor could be very much expressed as a dynamical system, and then potentially sort of reach out to that community which tend to have a very different formalism, but I think it's quite possible

469
01:08:30.090 --> 01:08:41.819
senate chamber: more generally, that sort of raises the issue. So sort of like you, you know. For, to take the example of the Covid model. You can sort of have behavior at scale there.

470
01:08:41.819 --> 01:09:06.710
senate chamber: And we tend to look at small scale behavior in Acta and look at very specific behavior and very specific measures. And one thing that we haven't, I think, explored as much is trying to look at the scalability of behavior. And in particular, as a computer scientist, you learn that the difference between 2 algorithms is not how fast they run on this example.

471
01:09:06.710 --> 01:09:09.439
senate chamber: example, right at how they scale with complexity.

472
01:09:09.439 --> 01:09:36.519
senate chamber: That's another way of looking at fundamental assumptions is looking at that scaling of behavior. For example, right? And again, that's sort of anticipating center a little bit. If you look at generative AI, they have those neural scaling laws that say, Okay, how is the behavior of a model going to scale with the amount of processing, power, training, data.

473
01:09:36.830 --> 01:09:52.410
senate chamber: reflection, time, or those kind of things. A number of parameters, and looking more systematically at emergent characterization of of models, might be another way to sort of take a formal view or a semi-formal view.

474
01:10:01.920 --> 01:10:05.320
senate chamber: No one else on the line has a hand up.

475
01:10:06.910 --> 01:10:08.090
senate chamber: I think we will.

476
01:10:09.330 --> 01:10:17.399
senate chamber: in the interest of I know not. All of the sessions are going to fit in there allotted time. We'll end this one a little bit early, and someone else will take it up the slack later. I'm sure

477
01:10:18.210 --> 01:10:42.019
senate chamber: that sounds good. A couple of practical notes. We have coffee on that side there, compliments of the conference for people who arrive late a reminder. If you go to the restroom, it's the restroom to the right there. Do not go where you came from, because it's taken by the Ohio Bar exam.

478
01:10:42.020 --> 01:10:51.509
senate chamber: and maybe we'll try to restart, let's say 5 min early. So at 1025, Eastern time for for the next session.

479
01:10:53.960 --> 01:10:57.049
senate chamber: Thank you again, everyone. Thank you. Everybody for comments.

480
01:11:03.540 --> 01:11:04.629
senate chamber: Yes, we can hear you.

481
01:11:05.300 --> 01:11:08.420
senate chamber: Ethernet is what we're actually on board.

482
01:11:10.950 --> 01:11:13.170
senate chamber: We're gonna get started with the next session.

483
01:11:18.240 --> 01:11:44.780
senate chamber: All right, thanks, everybody. With our second session of this workshop we have Mark Orr and his gaggle of philosophers that includes John Anderson, Kevin, Gluck, Frank Ritter, Andreas Stoko and Pete Peroli. But I don't think Pete is going to make it today. They'll be leading a discussion on the intricacies of models and theories and unification, and hopefully something about cognition. So Mark Orr, take it away.

484
01:11:44.940 --> 01:11:45.969
senate chamber: All right, thanks, Drew.

485
01:11:49.380 --> 01:11:54.570
senate chamber: I have my own timer who is Drew, so don't try to stop me. I'll stop myself.

486
01:11:54.790 --> 01:12:07.629
senate chamber: So the title of the talk today. So you guys know Centaur? Does anybody know the Centaur paper. Have you read it? Raise your hand if you have. It was on archive. It just came out in nature a week ago, and was put in all kinds of news sources. So it's hard to miss.

487
01:12:09.520 --> 01:12:31.809
senate chamber: So the title. This is about Centaur. The title of the talk is machine learning models of the statistics of human behavioral responses in experimental tasks. I was just trying to capture what I thought Centaur did. This is a group called the Intergenerational Centaur response group. I am just the organizer. So these thoughts are not my own. So if you want to blame anybody, don't blame me. You can blame these people drew

488
01:12:31.810 --> 01:12:47.450
senate chamber: Ken Ford, Kevin Gluck, Will Hancock, Crystal Lever Me, Pete Broe, Andrea Stoko and Frank Reeder and the University of Washington should be up here. I apologize, Andrea, but it has. Ihmc Cmu, Penn, State and University of Washington.

489
01:12:49.450 --> 01:12:57.470
senate chamber: I'll ask that. You defer questions until how does this work next slide?

490
01:12:58.690 --> 01:13:01.569
senate chamber: Nope, here we go. Okay?

491
01:13:01.780 --> 01:13:05.830
senate chamber: So defer questions until so the 1st part, I'm gonna just talk about Centaur and what

492
01:13:05.970 --> 01:13:22.119
senate chamber: at the very high level, what it is and what it does. So if you have questions during that period, if I'm not clear, please raise your hand, it'll take about 5 min or so, and then I'm going to go through some of our thoughts on Centaur and what it is and what it means.

493
01:13:22.420 --> 01:13:33.499
senate chamber: Then I'm going to ask that you don't ask questions, because what I'm going to do is give a very high level of this, and just give you the ideas. And the panelists are going to help flesh out the ideas. And then you guys can help flesh them out further.

494
01:13:34.200 --> 01:13:36.130
senate chamber: Okay, is that right?

495
01:13:36.880 --> 01:13:57.700
senate chamber: So Centaur is a pre-trained, large language model. It's based on Llama. I think the 3 billion parameter version. They have this data set called Psych 101. And the data set has 160 experiments from real humans. It accounts for 60,000 subjects approximately, and

496
01:13:58.010 --> 01:14:04.370
senate chamber: something like 10 million trials with responses that's compiled in this data set

497
01:14:04.700 --> 01:14:12.360
senate chamber: one of the key things about Centaur is that for it to work you have to take any experiment. I'm going to give you some examples of experiments in a moment

498
01:14:12.470 --> 01:14:39.860
senate chamber: you have to take the experiments, and they have to be translated into natural language in some way. That is a requirement, or it cannot work. So keep that in mind. It's a point of contention. I think so. I'll just take their language. Here. We introduce Centaur, a computational model that can predict and simulate human behavior in any experiment expressible in natural language. The any. There is a very important piece of that sentence. I would say. We'll debate about what predict and simulate mean if you want

499
01:14:40.500 --> 01:14:41.400
senate chamber: later.

500
01:14:41.830 --> 01:14:57.020
senate chamber: Here's a picture from the paper. So it takes this. So llama, the 3 billion parameter version. It freezes the weights, and it uses a low rank adapter for fine tuning, which means that

501
01:14:58.000 --> 01:15:17.729
senate chamber: it means that none of the weights when it optimizes on the new training data, which is the cycle, 101 data. None of the weights in the base model are optimized. Just the low, rank, adapter models, weights which are put on the outside. The way it works is they trained for the 10 million

502
01:15:18.180 --> 01:15:28.509
senate chamber: human responses. They trained one that box. So they showed one trial of those 10 million trials, one per. They went through the whole thing in a batch.

503
01:15:28.570 --> 01:15:51.550
senate chamber: and then they use error and just refined the fine tuning weights. So just to imagine how this works, I'm just going to fast forward for a second. Oh, oops, okay, so this is the context window. This is the input to Centaur for one of the data sets. I'm just going to walk you through this.

504
01:15:52.920 --> 01:16:06.800
senate chamber: the way it trains. You can just imagine skipping every every token in this case, a word, one at a time. And it's looking at what the prediction is. It has an error. It feeds that error back for the optimization procedure.

505
01:16:07.120 --> 01:16:13.819
senate chamber: But what they do is they throw out all of the error that's not related to a human response. So in this case here.

506
01:16:14.400 --> 01:16:21.029
senate chamber: you can. Just this is the go. No go task you can see near the bottom. It says, you see color one and press nothing.

507
01:16:21.320 --> 01:16:39.600
senate chamber: You see color 2 and press X and so forth. So while it's training, the only time it actually takes error into consideration is for the point where the human would actually respond. So this is the actual input. This is a translation of an experiment into natural language by some procedure that they came up with.

508
01:16:39.810 --> 01:16:41.640
senate chamber: which is, if anybody can guess.

509
01:16:41.760 --> 01:16:47.810
senate chamber: Does anybody know the procedure they use to transfer experimental paradigm into natural language?

510
01:16:49.090 --> 01:16:53.336
senate chamber: They do it by hand. It's just kind of a magical thing they do.

511
01:16:54.250 --> 01:16:55.799
senate chamber: I call it a procedure. Oops.

512
01:16:56.320 --> 01:16:59.490
senate chamber: I don't know what that is. So, anyway, that

513
01:16:59.690 --> 01:17:08.670
senate chamber: that is, this is the fine tuning. These are the details here. If anybody's interested, the newly added parameters are about a 10th of a percent of all the parameters in the model.

514
01:17:09.200 --> 01:17:28.500
senate chamber: I don't know if there's anything else to add here. So you can imagine you have 10 million responses. You go through those once fine tune, the system only on the component. You only take error from where the actual human behavior would be. Does that make sense to everybody? Roughly, any questions? Because I'm going to move on to

515
01:17:28.700 --> 01:17:33.339
senate chamber: more important parts. Yes, where do the response? Times come from?

516
01:17:35.890 --> 01:17:40.570
senate chamber: That's a good question. I think we'll pause on that one. But keep that or later.

517
01:17:41.340 --> 01:17:45.449
senate chamber: Okay, here's some of the tests.

518
01:17:45.730 --> 01:17:50.290
senate chamber: These are just I don't know 50 or so test separate categorization

519
01:17:50.580 --> 01:17:52.379
senate chamber: chicken. I don't know what that is.

520
01:17:53.020 --> 01:17:55.800
senate chamber: Did someone put check it in here.

521
01:17:56.270 --> 01:18:12.220
senate chamber: Frank. Frank, anyway. Forget the chicken stuff drifting forward and back. Digit span. Go. No go. Recent probes. We're going to give 2 examples of these in a minute, and so forth. So there's 160 tasks. You can look them up, but it's a nice variety of tasks, really.

522
01:18:12.610 --> 01:18:15.149
senate chamber: I don't know who that is.

523
01:18:15.370 --> 01:18:28.790
senate chamber: Is that Frank? Okay? So so the go, no, go test. This is the actual input, okay, I just want to walk you through this. If you were Llama.

524
01:18:29.280 --> 01:18:41.610
senate chamber: you would read this in this task. You need to admit responses. Blah blah! If you don't know the task. You see one of 2 colors, color one and 2 on the screen. You press X. When you see color one, and you press nothing when you see color 2.

525
01:18:42.730 --> 01:18:51.960
senate chamber: This is the input, it's for training and also for testing. So there's a 10% holdout on the 10 million responses that they use. So this is what the task looks like for Centaur

526
01:18:53.590 --> 01:18:54.969
senate chamber: to just keep that in mind.

527
01:18:55.150 --> 01:18:56.560
senate chamber: Here's another example.

528
01:18:57.370 --> 01:19:04.990
senate chamber: Recent probes you will be. You will repeatedly observe 6 sequences of 6 letters. You have to remember these letters before they disappear.

529
01:19:05.550 --> 01:19:15.839
senate chamber: Afterward you will be prompted with one letter you have to answer. Whether the letter was part of the 6 previous letters. If you think it was, you have to press C. If you think it was not press. Q.

530
01:19:16.500 --> 01:19:18.250
senate chamber: This is what Centaur was doing.

531
01:19:18.670 --> 01:19:21.889
senate chamber: You can walk yourself through it, imagining that your Centaur, if you'd like.

532
01:19:22.230 --> 01:19:23.929
senate chamber: I'll give you a second to do that.

533
01:19:25.590 --> 01:19:28.289
senate chamber: I don't know what it's like to be Centaur, but you could imagine it.

534
01:19:29.080 --> 01:19:31.120
senate chamber: Okay. So Centaur claims it was

535
01:19:31.290 --> 01:19:44.020
senate chamber: in the archive version in 2024 in the fall. It was the 1st true, unified model of cognition. I use the words that they used. Okay, just to be clear.

536
01:19:44.230 --> 01:20:09.119
senate chamber: And they softened the claim in 2025 in their nature paper from a couple weeks ago, so that that strong claim is in the conclusion is kind of hidden. But more generally they're saying that domain general models of cognition. This Centaur paper, this work is the next step or a next step, not the a next step towards a unified theory of cognition.

537
01:20:09.840 --> 01:20:14.819
senate chamber: So we can talk about what all that means. I don't know if anybody knows. But maybe we can figure it out.

538
01:20:15.690 --> 01:20:18.450
senate chamber: Okay, so we have 5 comments.

539
01:20:19.520 --> 01:20:23.299
senate chamber: 5 or 6 comments that we're making as a group.

540
01:20:23.709 --> 01:20:28.499
senate chamber: And there's others here, you guys have ideas we're happy to. So what we're doing is

541
01:20:28.680 --> 01:20:31.239
senate chamber: we're developing a response paper with this group.

542
01:20:31.400 --> 01:20:32.490
senate chamber: So

543
01:20:32.600 --> 01:20:42.929
senate chamber: we have a 1st draft amongst us. We're thinking about this. We're looking for ideas comments. You know, we're going to change this paper. That's part of the impetus for showing to you guys.

544
01:20:43.503 --> 01:20:47.640
senate chamber: So I'll go through and show you what our comments are, and then we can discuss it.

545
01:20:47.990 --> 01:21:00.030
senate chamber: So comment, one category mistakes. Centaur is so. They have this idea of a unified model of cognition, not a theory. What does that mean talk about that. We think it's really kind of a

546
01:21:00.380 --> 01:21:24.579
senate chamber: when I say category mistake. Oh, by the way, some of these interpretations, some of these ideas are still being vetted by the group that I'm working with. I'm showing you my version of it. There's other versions here. I think that you'll see from the panelists that diverge from this what I'm saying. So just keep that in mind. It's all provisional. So there's not really an architectural basis. They don't have the idea of a cognitive architecture as we might think of it.

547
01:21:26.120 --> 01:21:42.799
senate chamber: I think they use the word unified model, because to them it's a model that a single model that can work across tasks. That's what they mean. They're borrowing from a concept called integrated benchmarking from computational neuroscience, which they reference in their paper which is.

548
01:21:42.950 --> 01:21:46.020
senate chamber: take vision. For example, there's lots of

549
01:21:46.180 --> 01:21:58.489
senate chamber: mechanistic computational models in vision. Lots of data share the data and start looking at how the models can apply across the different data sets. That's their idea. They're using the same idea here.

550
01:21:58.780 --> 01:22:02.350
senate chamber: So that's what they mean, in my view, by unified model.

551
01:22:03.170 --> 01:22:07.040
senate chamber: This is a graphic of what I mean.

552
01:22:07.140 --> 01:22:34.039
senate chamber: So you can see in the middle task space. You have higher complexity, lower complexity. Think of higher complexity as something in the real world with time, constraints, and flexibility, lower level tasks and perceptual tasks. As examples. Centaur is a model that applies in their version with the caveat that it's in natural language. It applies to less complex tasks. It's a single model. Architectures have.

553
01:22:34.190 --> 01:22:41.219
senate chamber: as you all know, some kind of theory which is an architecture which then generates models which can span lots of different tasks.

554
01:22:41.460 --> 01:22:46.989
senate chamber: Another version of this, so we call this category mistake, because the structures aren't the same.

555
01:22:49.140 --> 01:22:56.190
senate chamber: So this is Andrea Stoko's version at the top. What he points out is that there's some models that

556
01:22:56.330 --> 01:23:06.649
senate chamber: there's some things humans can't do. It's like they don't have unlimited memory. They don't process certain things. They can't process, or certain things under certain time constraints.

557
01:23:07.336 --> 01:23:09.280
senate chamber: There's bounds on humans.

558
01:23:09.780 --> 01:23:21.879
senate chamber: Centaur does not care about that. You could give it really crazy data. You could give it data with people who just don't understand the tasks, and it would actually probably model them pretty well. It would model tasks that humans can't do very well because it doesn't care.

559
01:23:22.800 --> 01:23:32.970
senate chamber: So that's a category mistake. We're trying to figure out what unified model means, what unified, how it relates to unified theory. We don't think they're actually going in the same direction.

560
01:23:33.680 --> 01:23:44.420
senate chamber: Number 2, and we'll talk a lot about this Newell's test. They do invoke Newell's test criteria for models of cognition or theories of cognition.

561
01:23:44.580 --> 01:23:53.259
senate chamber: We offer some criteria, some comments on all 12 criteria, and I'll let the panelists slash that out. But I will point out what Centaur said.

562
01:23:53.370 --> 01:24:13.900
senate chamber: This is in the nature version in the supplementary material, together with his call for unified theories of cognition, Newell outlined a set of criteria that, unified that a unified computational model should fulfill. Centaur is the 1st model to satisfy the majority of the criteria. This is in nature. Okay.

563
01:24:14.280 --> 01:24:32.219
senate chamber: and this is their table. It's really good at everything except for, of course, a rise through evolution, biological evolution, and acquiring capabilities through development which is related, and then the gray one. I guess that means undecided which is, exhibit self-awareness and a sense of self. So we'll have to wait on that one.

564
01:24:32.700 --> 01:24:33.659
senate chamber: Maybe it's true.

565
01:24:34.360 --> 01:24:43.409
senate chamber: comment number 3. No one likes this argument in my group. I'll just tell it to you, anyway, because I want to see if anybody else likes it, and

566
01:24:43.520 --> 01:24:56.140
senate chamber: I think it's useful to talk to friends and family about. So it's called. It's the analogous to the large language model consciousness argument. Argument. Okay, so do you guys remember the there was an engineer at Google who thought that?

567
01:24:56.390 --> 01:24:57.850
senate chamber: Yeah.

568
01:24:57.990 --> 01:25:04.460
senate chamber: So this person thought it was conscious, warned the manager, that this thing's conscious. We need to give it rights and things.

569
01:25:04.890 --> 01:25:09.749
senate chamber: And does anybody know why they're trying to give AI rights. They're not

570
01:25:10.350 --> 01:25:21.179
senate chamber: well, they're going to mint people. They're going to mint people. So no, I'm just kidding. It's a joke. It's a conspiracy theory. We'll say that for for drinks. No, but seriously, so this. So this idea.

571
01:25:21.460 --> 01:25:22.960
senate chamber: what's his name? Cock?

572
01:25:23.520 --> 01:25:35.769
senate chamber: The neuroscientist Christoph Koch. Okay, he has this idea. Well, that's just a deep fake. It's totally deep. Fake because think about it. What is Centaur? What the large language models are trained on

573
01:25:35.900 --> 01:25:46.960
senate chamber: all kinds of literature. What does literature have in it? Well, if I wanted to convince you, I was conscious and I wasn't, I would read back to you some passages from Tolstoy, and you'd probably think I was really conscious.

574
01:25:48.280 --> 01:26:01.729
senate chamber: so it's just in there, right? And the argument's the same here. If you could tell someone in the street and say, Look, it's they've read all the novels that you've read, and all the novels that you'll never read, and all the novels that none of your friends will ever read, because they can read all the novels.

575
01:26:01.960 --> 01:26:04.270
senate chamber: It's not surprising they sound conscious.

576
01:26:04.550 --> 01:26:17.099
senate chamber: That's the argument. It's the same thing here. This model has taken all of this data in from the experiments, and it's capturing it. How different do you think the people are in that training set. Are they vastly different on these tests?

577
01:26:17.690 --> 01:26:25.669
senate chamber: Hmm, don't know, anyway, that's our comment. Number 3 is, the illusions are illustrations, illusions, argument

578
01:26:26.650 --> 01:26:29.420
senate chamber: number 4. This one's a little more serious.

579
01:26:29.590 --> 01:26:31.439
senate chamber: This is my favorite in the group.

580
01:26:31.730 --> 01:26:32.680
senate chamber: So

581
01:26:32.850 --> 01:26:37.609
senate chamber: this table here, so I can't. I can't walk over here. Is that true? Because you can't hear me.

582
01:26:38.000 --> 01:26:43.469
senate chamber: I usually walk around. That's why I'm gripping this. So I don't go crazy.

583
01:26:43.740 --> 01:27:01.110
senate chamber: So this table here is a 4 by 4, a 2 by 2 on the one axis we have experimental version, and what do I mean by that? Now? Remember, they took 160 experiments, and by some procedure, they translated them into natural language.

584
01:27:01.220 --> 01:27:12.389
senate chamber: So we have the Centaur version, which is the translated version, and we have the original version, which is whatever that is. Now, we don't have to define that, but it is the original, because they at least took it as input. So we'll just call it the original.

585
01:27:12.800 --> 01:27:18.040
senate chamber: and then the other axis. We have experimental subject. We have centaur and human.

586
01:27:18.850 --> 01:27:26.950
senate chamber: So we claim that Centaur, the group missed an opportunity which is notice that

587
01:27:27.190 --> 01:27:34.370
senate chamber: the human can do either task. They obviously did the original, because that's the data. But they can also do the Centaur task.

588
01:27:34.510 --> 01:27:38.620
senate chamber: They can read that stuff and learn from it if they want to, and then perform

589
01:27:39.300 --> 01:27:41.720
senate chamber: Centaur cannot do the original task.

590
01:27:41.990 --> 01:27:49.319
senate chamber: it can do the Centaur task. It's designed for that. So the point is, what's in red here? They missed the opportunity.

591
01:27:49.640 --> 01:27:54.879
senate chamber: They didn't give humans the same task. They claim it's the same task. They didn't give humans the same task.

592
01:27:55.070 --> 01:27:56.300
senate chamber: So you have to wonder.

593
01:27:56.540 --> 01:28:05.899
senate chamber: Well, it's a bullshit argument, right? Of course they didn't give humans that task. It would be ridiculous to read 10 million responses and then learn from that and then do things.

594
01:28:06.230 --> 01:28:10.370
senate chamber: But that's exactly the point, because it's not the same thing.

595
01:28:12.650 --> 01:28:21.719
senate chamber: Comment number 5 claims of neural alignment are overblown. Andrea Stocco and Drew Cranford are responsible for this one, and I believe it, and I'm going to leave it at that.

596
01:28:22.000 --> 01:28:36.890
senate chamber: So this one here centaur is non-mechanistic and atheoretical. This underpins probably most of the discussion today. It's implicit, I think, for a lot of people. Once we talk about it. So we don't need to go into all the details. But I will just say.

597
01:28:37.830 --> 01:28:49.079
senate chamber: keep in mind Centaur's role. If it is a theory, 2 things. If the predictions are, let's just imagine the predictions are really poor. It doesn't do a good job. What would you do next?

598
01:28:49.250 --> 01:28:51.230
senate chamber: That's the question. If it's a

599
01:28:52.160 --> 01:28:57.770
senate chamber: you have to think about that. What insights would you glean if it didn't do well, or if it doesn't do well in some tasks.

600
01:28:57.990 --> 01:29:03.079
senate chamber: what do you do? Do you change aspects of it? Which of the well, so it's

601
01:29:03.200 --> 01:29:08.550
senate chamber: it's a 10th of a percent of 3 million. So it's about 500,000, or something.

602
01:29:08.940 --> 01:29:10.649
senate chamber: which parameters do you change?

603
01:29:11.750 --> 01:29:16.789
senate chamber: And if the predictions are good which they claim they are, what do you do next?

604
01:29:16.950 --> 01:29:21.739
senate chamber: Is there? Does it make new predictions? They have some kind of new predictions in the

605
01:29:21.870 --> 01:29:24.420
senate chamber: paper. I haven't evaluated them, but

606
01:29:24.670 --> 01:29:28.859
senate chamber: we have to think through those questions is my point. I don't think they were thought through yet.

607
01:29:29.580 --> 01:29:39.520
senate chamber: We call it an epistemic black box stole that from Dan Dennett. I stole other stuff from him, too. I don't acknowledge that on purpose. It might not be the right term. But I just like how it sounds.

608
01:29:39.930 --> 01:29:50.479
senate chamber: So. Oh, okay, 3 min, 2 min left alternative trends you probably know about. So there's work trying to build cognitive models from scratch. We're doing some work for

609
01:29:50.700 --> 01:30:09.030
senate chamber: for Darpa with Christian and some others in the room. But also there's work here where they have guided computational cognitive models. There's other people using cognitive architectures. The term they use the term cognitive architectures, but they're different approaches. There's a guy at Cmu and Bethune.

610
01:30:09.570 --> 01:30:18.029
senate chamber: So, and generative agents are ways of kind of simulating agents for purposes of getting behavior out of agents that aren't humans.

611
01:30:18.460 --> 01:30:27.420
senate chamber: There's a lot of trends going forward. And my idea is this, and this is a really contentious idea. And it's new. I came up with it yesterday, so it may not be any good.

612
01:30:27.590 --> 01:30:32.210
senate chamber: But Alphafold is an amazing kind of scientific

613
01:30:32.580 --> 01:30:37.100
senate chamber: approach to a scientific problem and a technological approach.

614
01:30:37.300 --> 01:30:38.739
senate chamber: And I think that

615
01:30:38.930 --> 01:30:55.109
senate chamber: I'm just going to recommend that the community as a whole in general, if you're studying intelligence in any way or cognition should start thinking about instead of saying, well, things are possible in principle, but then, you know, in fact, it's not possible at all. That's something you learn over time.

616
01:30:55.110 --> 01:31:10.909
senate chamber: we should start looking at things that are impossible in principle, and actually see if they're possible. In fact, I think we need to do something like this. For example, my stick on what is cognition is an evolutionary biological one. Evolutionary biology has changed quite a bit

617
01:31:11.370 --> 01:31:26.900
senate chamber: whenever you're studying any species, including humans. And I think everybody agrees with this. But I don't think it's taken seriously enough in general in the sciences is that we're looking at 2,500 million years of evolution whenever you study humans, including other species.

618
01:31:28.590 --> 01:31:33.889
senate chamber: so there's a lot of work out there. And I think that approaches like, I'm not saying we use alphafolds approach

619
01:31:34.020 --> 01:31:56.580
senate chamber: in terms of the actual technology. But they really pushed mixing machine learning, instruction and scientific structures into one kind of system. And I think that we need to push forward and really try to look at cognition from different perspectives over the next 20 years. And it would take a corporation and millions and millions and millions of dollars.

620
01:31:56.690 --> 01:32:02.560
senate chamber: That is a like, oh, Frank, that's it. So right on time.

621
01:32:03.150 --> 01:32:03.820
senate chamber: Cool

622
01:32:08.620 --> 01:32:11.790
senate chamber: any quick questions to me before we start the panel.

623
01:32:12.050 --> 01:32:13.349
senate chamber: Sorry, Drew.

624
01:32:16.090 --> 01:32:17.430
senate chamber: Okay, good.

625
01:32:19.160 --> 01:32:29.599
senate chamber: Thank you, Mark. And at this time I'll invite the panelists to the front to the podium. And mark. You can have a seat there with them at the table.

626
01:32:38.550 --> 01:32:47.360
senate chamber: Frank and online, we have Kevin Bluck and Andreas Stoko.

627
01:32:48.186 --> 01:32:54.110
senate chamber: I don't believe Pete Peroli was able to make it today. But if you are here, Pete, please speak up.

628
01:32:55.780 --> 01:32:56.800
senate chamber: hey, OP!

629
01:32:58.530 --> 01:33:08.969
senate chamber: I don't have any structured time segments for anybody in this session. So. But I will cut you off if you start speaking too late.

630
01:33:10.820 --> 01:33:14.299
senate chamber: Hey? Did you remember your request? Yeah.

631
01:33:14.670 --> 01:33:19.759
senate chamber: I think it'd be more useful if this wasn't presented as my material, that the panelists just

632
01:33:19.920 --> 01:33:38.260
senate chamber: the last session. It was clear that there was questions for Joe, and addressing right. I don't want to answer questions like that in the sense of I'm not here, you know, as an authority, I'm just here as a person just part of the group. So just want to make that clear. You can moderate that as you start attacking me.

633
01:33:42.406 --> 01:33:54.070
senate chamber: I don't have any again times for anybody to speak or anything, but I will make sure to go through the panelists to make sure everybody does get a chance to

634
01:33:54.440 --> 01:33:56.469
senate chamber: same points that they do have

635
01:33:58.530 --> 01:34:03.550
senate chamber: So if who wants to start us off right? Do you have anything you want to say?

636
01:34:05.780 --> 01:34:07.639
senate chamber: Well, oh, sorry.

637
01:34:07.790 --> 01:34:13.079
senate chamber: and please turn your mics on at the podium when you are speaking. Oh, are they on?

638
01:34:13.790 --> 01:34:14.892
senate chamber: Oh, how's that?

639
01:34:17.290 --> 01:34:20.700
senate chamber: Oh, wait! Sorry! That's not. That's just something I was watching.

640
01:34:26.660 --> 01:34:31.370
senate chamber: So here's some comments. Can you hear me online? Is that good enough?

641
01:34:31.660 --> 01:34:34.130
senate chamber: I can't hardly put that in my mouth.

642
01:34:34.810 --> 01:34:39.370
senate chamber: So here's some quick notes, right? I thought this was worth taking. Seriously.

643
01:34:39.480 --> 01:34:47.740
senate chamber: I think there's some serious flaws, and I think it's useful for the whole room and the community to have paid attention to this paper, and I think that's part of what the

644
01:34:47.840 --> 01:34:57.049
senate chamber: role this symposium is. So this this system here. But what can we learn? And I learned a couple of different things. I'm trying to get into that paper.

645
01:34:57.800 --> 01:35:01.359
senate chamber: Sorry you're the first.st I'm the last.

646
01:35:01.680 --> 01:35:14.619
senate chamber: and I'm not chicken to put it in so large, direct, cooperative efforts. It's a. You look at the list of people. They're sort of around the world, and they're all working together. And it's it's

647
01:35:14.720 --> 01:35:17.780
senate chamber: like how I thought science was done when I was a kid.

648
01:35:18.190 --> 01:35:36.030
senate chamber: They're doing multitask modeling. They're modeling more than one task. I've called for it with Fernand Gobay, and I think it's a lovely thing. But how to join this is not clear. And where is it going like you started to pick it back, mark of like, you know. Where do you go next?

649
01:35:37.570 --> 01:35:43.799
senate chamber: There's a large unified database of human behavior which is lovely.

650
01:35:44.100 --> 01:35:59.890
senate chamber: It's the largest that we are aware of. It's the largest I'm aware of. I think their paper notes that it's, you know, rivals the size of physics databases. I think that's how they got into the journal. But that guy up there that said about reaction times.

651
01:36:00.400 --> 01:36:01.990
senate chamber: They throw them out.

652
01:36:02.730 --> 01:36:04.300
senate chamber: They're not included.

653
01:36:05.100 --> 01:36:14.989
senate chamber: and I don't think I saw a talk in the last 3 days that only looked at behavior. Nearly all of the talks at this conference looked at reaction times.

654
01:36:15.640 --> 01:36:34.349
senate chamber: And so you can't see learning on a response time. You can only see changes in behavior, and I think that's I can't quantify it, but it's a thinner result. But you've got to average over 5 or 10 or 100 to see how that changes, and it's a much slower change in response time.

655
01:36:34.690 --> 01:36:36.849
senate chamber: And they threw out vision.

656
01:36:38.120 --> 01:36:46.810
senate chamber: And so if you get a stroop task. You get the red and blue, the blue and blue. Pay attention to which order it is, and you're done

657
01:36:48.240 --> 01:36:54.940
senate chamber: so so there's some things they can't all do so in many ways the data set is useless.

658
01:36:56.030 --> 01:36:58.519
senate chamber: but in some ways it could be fixed. Perhaps.

659
01:36:59.530 --> 01:37:01.689
senate chamber: So I'm I'm torn by that.

660
01:37:02.390 --> 01:37:05.880
senate chamber: It can build models really quickly.

661
01:37:06.390 --> 01:37:15.629
senate chamber: and there is some work by glock and Laird and the group there that talks about how to build actar models and presumably soar models through task descriptions.

662
01:37:16.000 --> 01:37:19.780
senate chamber: But I think it's essentially categorical regression.

663
01:37:20.130 --> 01:37:22.549
senate chamber: and I don't know what it means.

664
01:37:23.500 --> 01:37:30.749
senate chamber: But anyhow, there's a guy named Clayton Lewis who's not here who has a book out just recently that'll be in the references.

665
01:37:31.324 --> 01:37:36.700
senate chamber: You could get a copy from Clayton in your library, or perhaps from me.

666
01:37:37.470 --> 01:37:43.310
senate chamber: where he talks about. What can we learn from these large language models if we treat them as a psychology theory.

667
01:37:44.760 --> 01:37:52.319
senate chamber: which is great stuff. So here's my, oh, so here's my part of my answer to them, and they also use this answer to have a scorecard.

668
01:37:52.430 --> 01:37:58.950
senate chamber: And in a textbook that I've got 7 tenths done on how to model.

669
01:37:59.440 --> 01:38:09.330
senate chamber: I think you want to call your shot, Newell said. Science is what techniques you will build that descendants will have to use. I think we're starting to use scorecards now.

670
01:38:09.960 --> 01:38:20.979
senate chamber: and the picture here is of Babe Ruth. He's calling his shot where he's going to hit the ball. Here's an example, Scorecard, the 1st one I ever saw by Bonnie John, and you can see.

671
01:38:21.160 --> 01:38:23.109
senate chamber: No, we can't. Oh, you can't see.

672
01:38:24.822 --> 01:38:29.670
senate chamber: Oh, you didn't see any of this.

673
01:38:30.100 --> 01:38:35.250
senate chamber: We can't say. If we can't read, it's a word.

674
01:38:36.140 --> 01:38:43.139
senate chamber: Oh, okay. So I've shared the wrong. No, no, it's on the screen. It's just small, small.

675
01:38:43.350 --> 01:38:48.139
senate chamber: Well, I can't make I? Oh, okay, I can make it bigger zoom in. Yes.

676
01:38:49.490 --> 01:38:51.550
senate chamber: what I'm on to mark.

677
01:38:51.690 --> 01:38:57.499
senate chamber: How can I zoom in if I'm on zoom. Is that better? Yes. Oh, okay.

678
01:38:58.450 --> 01:39:08.859
senate chamber: So here's the 1st scorecard I saw, and they have a scorecard. But here's my version of their scorecard. Does it match the data, you know. Sort of.

679
01:39:09.020 --> 01:39:16.869
senate chamber: is it easy to build, absolutely easy to extend? Sort of easy to understand? I don't think anybody understands it

680
01:39:17.500 --> 01:39:20.709
senate chamber: make new predictions? I don't think so.

681
01:39:21.020 --> 01:39:27.149
senate chamber: Does it learn? No, and I think that's a gain it loses for me because of that.

682
01:39:27.380 --> 01:39:30.650
senate chamber: Does it operate and predict in real time. No.

683
01:39:31.100 --> 01:39:41.249
senate chamber: Does it make errors absolutely great stuff? And you could then start to have a scorecard, and we could start to compute what we like and don't like on a scorecard basis.

684
01:39:41.710 --> 01:39:50.770
senate chamber: So then, here's my insights, which I think is the point of work. Anyhow, there's some really nice things about this work, and we're going to have to read and reread this paper.

685
01:39:51.060 --> 01:39:56.320
senate chamber: Does this, Nick? Nor Nick corrected as is Nick Einal false

686
01:39:57.800 --> 01:40:01.539
senate chamber: right? It's not even. It's not right. And it's not even wrong.

687
01:40:02.420 --> 01:40:09.569
senate chamber: because it's hard to show that it's wrong. And this is a famous physicist talking about a really bad physics theory.

688
01:40:10.050 --> 01:40:12.460
senate chamber: but we're gonna have to pay attention to it. It's not famous.

689
01:40:12.690 --> 01:40:20.280
senate chamber: I think we now need. And this is based on listening to your talk. We need to worry about worry about lying on criteria tests

690
01:40:20.610 --> 01:40:27.699
senate chamber: because some of the things they claim I don't agree with at all. They say it does that, and I don't think it does that.

691
01:40:28.440 --> 01:40:30.270
senate chamber: And one thing I've learned in the last

692
01:40:30.410 --> 01:40:34.160
senate chamber: 12 years is you can say anything you want.

693
01:40:34.990 --> 01:40:37.470
senate chamber: and you just don't get really called out

694
01:40:38.650 --> 01:40:41.300
senate chamber: in public, at least in in some places.

695
01:40:41.620 --> 01:40:51.919
senate chamber: And I'd like to think of some responses we could think about this. Now we've got a lot of brain power, you know. We could ask Centaur some new questions like, how many times will you touch a burning stove?

696
01:40:53.150 --> 01:41:01.229
senate chamber: You know the answer is sort of once, but you know it might say more, and if we run it through the Milgram shopping experiment.

697
01:41:01.480 --> 01:41:04.270
senate chamber: will it? Will it? Will it kill people.

698
01:41:04.610 --> 01:41:09.190
senate chamber: you know. Will it run the high shocks, or will it be, you know, more attentive?

699
01:41:09.450 --> 01:41:12.799
senate chamber: And can it do the the strip task?

700
01:41:13.340 --> 01:41:17.770
senate chamber: Can it do something like the Klingon laser bank task where you're doing problem solving.

701
01:41:18.750 --> 01:41:28.199
senate chamber: You know. You can give it the crime, you know. Give it well, you have to put in words, though. Can it do the traveling salesman problems which we know people can do approximately?

702
01:41:29.520 --> 01:41:30.980
senate chamber: Can it play chess?

703
01:41:31.590 --> 01:41:34.609
senate chamber: And I need to go off and replay Moose's book.

704
01:41:34.950 --> 01:41:39.800
senate chamber: So those are my prepared comments, and I also referenced Zonker.

705
01:41:44.620 --> 01:41:46.840
senate chamber: So thank you. Thanks, Greg.

706
01:41:47.370 --> 01:41:49.469
senate chamber: Anybody have any comments or questions on that.

707
01:41:58.080 --> 01:42:01.770
senate chamber: We have one online, and then I'll come back to Alex.

708
01:42:03.037 --> 01:42:07.060
senate chamber: Nancy. You can unmute yourself and ask your question.

709
01:42:08.450 --> 01:42:13.899
Christopher Dancy: I guess this is more of a comment, and I guess part of the question. But

710
01:42:14.510 --> 01:42:20.140
Christopher Dancy: and more of a come. Hello! I feel I feel so, professors saying that so.

711
01:42:21.180 --> 01:42:25.047
Christopher Dancy: Frank, I mean, I think I know the answer to this right?

712
01:42:25.520 --> 01:42:33.549
Christopher Dancy: What I'm wondering is for you. What are the recent examples of kind of

713
01:42:33.890 --> 01:42:36.970
Christopher Dancy: engineering pipelines that allow us

714
01:42:37.460 --> 01:42:50.530
Christopher Dancy: as cognitive scientists to be able to easily implement the models we want. Because one thing I think I'm constantly seeing right. And one of the reasons why, for example, Llms can scale and make all of these

715
01:42:50.870 --> 01:42:54.319
Christopher Dancy: all of these claims, and

716
01:42:54.620 --> 01:43:22.609
Christopher Dancy: we can get it. If I was there, it would be a good talk over. Be right on all the problems, with all the claims that many element systems make the folks who are developing them. But one of the things they can do is they can scale because there's some people who can use them because they build communities, and they build engineering pipelines that allow them to be used in these different ways. And so I'm wondering, Frank, if you have seen recent stuff that is useful to point to for people in the community for kind of pipelines that allow us to develop models and ways that we

717
01:43:22.610 --> 01:43:29.790
Christopher Dancy: can scale and think about limiting our theories. Or if you just have any comments kind of related to the creation of those

718
01:43:29.850 --> 01:43:30.940
Christopher Dancy: pipelines.

719
01:43:31.240 --> 01:43:38.140
senate chamber: So I'm writing out notes here. One. This work by Glock and laird and contributors to their edited book

720
01:43:38.530 --> 01:43:43.929
senate chamber: on Interactive task learning, I think is like would be really helpful in all kinds of ways.

721
01:43:44.080 --> 01:43:58.750
senate chamber: and going back and continue looking at Pdp plus plus where it builds its models through regression of a type, high level behavior, representation languages. And then there was a triple AI workshop

722
01:43:58.970 --> 01:44:07.419
senate chamber: where people tried to get large language models to write their soar and actar code, and they it only worked on declarative knowledge.

723
01:44:07.850 --> 01:44:18.439
senate chamber: which was really sad, because it would have been nice to have someone help us with our homework. But one of my students, Xu was Bosch and

724
01:44:18.820 --> 01:44:26.810
senate chamber: Alessandro. Injected actar traces into large language models. So maybe you could co-opt

725
01:44:28.140 --> 01:44:35.170
senate chamber: these large language models and tell them how to think those are my off. The cuff comments.

726
01:44:36.930 --> 01:44:37.510
Christopher Dancy: Good.

727
01:44:39.780 --> 01:44:57.590
Kevin Gluck: Let me let me just jump in and say that I I really appreciate the nod to the interactive task learning book and and the pointer to it, and I hope that many people find that to be, you know, interesting and inspirational, and all of that. But I do want to clarify

728
01:44:57.720 --> 01:45:01.230
Kevin Gluck: that in that book we are not providing

729
01:45:02.156 --> 01:45:19.319
Kevin Gluck: end to end computational pipeline solutions for achieving interactive task learning. That book is based on the content generated during a stringman forum, which is a sort of 5 day long event.

730
01:45:19.750 --> 01:45:22.709
Kevin Gluck: sponsored by the Stringman Foundation.

731
01:45:22.960 --> 01:45:34.030
Kevin Gluck: that brought together a lot of people from different disciplines to talk about the idea concept, vision challenges barriers to progress and and things

732
01:45:34.210 --> 01:45:40.999
Kevin Gluck: out of multiple scientific communities that would be relevant to know about in interactive test learning.

733
01:45:41.350 --> 01:45:49.269
Kevin Gluck: But nowhere in that book is there actually a solution to that to that problem. And so I just wanted to clarify.

734
01:45:49.440 --> 01:45:54.210
Kevin Gluck: So that so the people have the right expectation. If they're going to look at the book, it's all.

735
01:45:54.230 --> 01:45:55.670
senate chamber: You're pointing, though.

736
01:45:57.600 --> 01:46:14.279
Kevin Gluck: Okay, well, yeah, I appreciate that. Thank you. I just wanted people's expectations to be clear. And I'll also point out that the entire book is available online open access through the Stringman forum website. So you can download any any one of the individual chapters you'd like for free.

737
01:46:20.020 --> 01:46:29.000
senate chamber: All right. Thanks. We'll continue through the panel before getting to any more questions, and then we'll come back.

738
01:46:29.290 --> 01:46:34.969
senate chamber: Okay, unless it's important. Do you have it. It was a point of clarification for

739
01:46:35.150 --> 01:46:46.249
senate chamber: so so sorry. Sorry to jump in but I I recall you stating in your overview that

740
01:46:46.350 --> 01:46:48.219
senate chamber: sent to our home gates.

741
01:46:49.090 --> 01:46:50.490
senate chamber: Response time.

742
01:46:50.680 --> 01:47:13.340
senate chamber: And in the previous example that we saw clearly emits some response time data. They claim it operates in real time you saw from their checklist. You claim it doesn't, I think I probably agree. But can you very briefly speak to why, you seem to be disagreeing that they do it.

743
01:47:13.590 --> 01:47:15.169
senate chamber: You've looked at the traces

744
01:47:16.880 --> 01:47:28.100
senate chamber: which ones they're they're not predicting response times. No? Right? That's right. Yeah. And the data that goes in is not response time. They show response times in the picture. But there's

745
01:47:28.910 --> 01:47:35.479
senate chamber: but that's not yet. They're actually generating the optimization for the fine-tuning doesn't

746
01:47:35.600 --> 01:47:42.780
senate chamber: optimize on response time. In other words, those tokens 3 0, on the same.

747
01:47:43.070 --> 01:48:04.530
senate chamber: Oh, I came. So they don't optimize on that part of the training. So the only optimization that actually counts is when there's a response of the button press. Oh, so they don't even like the noting the tokens. That's right. Exactly. So just imagine

748
01:48:04.530 --> 01:48:17.230
senate chamber: you hit X in 350 ms. You get error for all those tokens. But it throws out all the error unless it's the response itself. Okay, all right. So just think of it as a token, that doesn't matter. Yeah.

749
01:48:17.520 --> 01:48:21.449
senate chamber: And the and the data that I looked at, I didn't go through all the data.

750
01:48:21.800 --> 01:48:26.850
senate chamber: It just had the responses from the humans, not their response time.

751
01:48:27.630 --> 01:48:33.610
senate chamber: So they very carefully curated that data set. And then, as far as I can tell, they did not save that column.

752
01:48:34.180 --> 01:48:42.579
senate chamber: Incredible, but we should check that, you know, and say not just one, but all of them. They do on some. Yeah, I mean, it's there, but it's not used for the

753
01:48:42.720 --> 01:49:07.780
senate chamber: well when I looked at it, but I don't know which one. I looked at. My understanding is those things you see in brackets. That's the only thing they're filling in is what's sitting in those angular brackets. Everything else is. It's there, and they call it a reaction time experiment. But all they're doing is predicting what the actual button press is.

754
01:49:08.210 --> 01:49:14.349
senate chamber: Yeah, I agree with that. So it could, it could do really well at press the button when the light's on.

755
01:49:20.120 --> 01:49:20.930
senate chamber: Thanks.

756
01:49:24.620 --> 01:49:30.539
senate chamber: All right, we'll continue through the panels. John, did you have any points you wanted to make.

757
01:49:32.760 --> 01:49:45.790
senate chamber: Well, I can make a few comments. 1st of all, I'm an interruption of all the authors of the paper, I'm not one. And going back to the comment.

758
01:49:47.240 --> 01:49:59.270
senate chamber: sort of things are useful, if not true. I think this is a lot to be impressed by this particular paper, and in some sense it isn't

759
01:49:59.400 --> 01:50:02.790
senate chamber: misinterpreted. It's it's going to serve a useful function.

760
01:50:05.090 --> 01:50:15.939
senate chamber: I mean, I'm jealous of a hundred 50 tasks. All that data. It's what what I've always aspired to do, and, in fact, are never come close.

761
01:50:17.600 --> 01:50:23.200
senate chamber: so I think we should give credit where credit's due.

762
01:50:23.460 --> 01:50:25.300
senate chamber: One could hope for more.

763
01:50:25.550 --> 01:50:39.620
senate chamber: I hope for the day that the model is just going to spit out and act. The Centaur is just going to spit out and act our model for the task that it's doing. And then we can sort of really understand what's going on.

764
01:50:41.190 --> 01:50:45.440
senate chamber: Yeah, I just really have one additional comment to make, which is

765
01:50:46.250 --> 01:50:50.809
senate chamber: quite a while. It's almost a missed opportunity in that paper.

766
01:50:51.650 --> 01:50:55.849
senate chamber: because what they're doing is what we would call model tracing.

767
01:50:56.660 --> 01:51:04.869
senate chamber: They're taking the history of that a particular individual has done up until the point of the next button press.

768
01:51:05.150 --> 01:51:14.629
senate chamber: and using that as part of their information to predict what button. That person will press next. So they're tracking individuals

769
01:51:14.840 --> 01:51:17.270
senate chamber: and it

770
01:51:18.400 --> 01:51:46.900
senate chamber: and model tracing is a really useful technology and maybe not very different from what they're trying to predict in a lot of educational applications. There's a data mining community that's in education that's really interested in trying to follow what a student knows and what a student doesn't know and kind of predict their future. So I think in some ways, if they cast themselves as having solved.

771
01:51:47.000 --> 01:51:54.029
senate chamber: having a great solution to a big fraction of the model tracing problem. Maybe we don't really understand what the students are doing.

772
01:51:54.330 --> 01:51:59.589
senate chamber: But we're getting the predictions that you would want in those kind of educational situations.

773
01:51:59.730 --> 01:52:10.739
senate chamber: I think in some ways moving in that direction. I suspect things of educational data mining are that would be a real sort of

774
01:52:11.270 --> 01:52:20.049
senate chamber: kind of real right now, contribution of that particular approach, and and it is the fact that they do take novel tasks

775
01:52:20.310 --> 01:52:33.789
senate chamber: that that model hasn't been trained on kind of follow that model tracing approach and kind of predict the next thing so it could be something like that could be really useful in the context of educational applications.

776
01:52:37.180 --> 01:52:40.570
senate chamber: All right. I have a quick question for John, if I may before.

777
01:52:41.080 --> 01:52:57.100
senate chamber: So you say we are. Actar has not come close, and I don't know how you mean that, but I would disagree generally. So actor has certainly been used to model. I would guess hundreds of tasks. I don't know that you're in a thousand, certainly hundreds.

778
01:52:57.140 --> 01:53:11.610
senate chamber: so you could imagine, you know. Take an Avdar model, an application load all these hundreds of models in there and say, Oh, if you're doing stroop, call that model, if you're doing this, call that model

779
01:53:11.830 --> 01:53:26.109
senate chamber: that would be, I would argue, and that the question is, how unified? Right? So it's unified in the sense, in the neural sense, of those models. Use the same mechanisms of the architecture, so they're unified in that extent.

780
01:53:26.110 --> 01:53:42.369
senate chamber: They're not as unified as they could be, because, oh, is this model of air traffic control uses the list learning model? Maybe it should. But it doesn't right, because all models are modular pieces, so it's maybe not as unified as we could, but at least somewhat unified.

781
01:53:42.450 --> 01:54:00.620
senate chamber: Do we know how unified satar is. Do you know, what do we know what the overlap in those weights is between doing one task versus the other? We don't. So I think that it raises some interesting question of what unified means, but I would argue that we've certainly done that at least meant that

782
01:54:00.780 --> 01:54:05.260
senate chamber: that benchmark of applying the architecture to hundreds of models.

783
01:54:05.430 --> 01:54:22.179
senate chamber: Maybe I don't understand all. What centar did the fact that we can model hundreds of tasks reflects probably hundreds of years of people working on and developing those individual models.

784
01:54:23.025 --> 01:54:23.730
senate chamber: So

785
01:54:23.950 --> 01:54:31.399
senate chamber: not that that wasn't pointed out. But it wasn't a great group effort at some level of getting to where they are in center.

786
01:54:34.650 --> 01:54:46.569
senate chamber: but my my impression is that that there is kind of 1 1 centaur system that's been trained, and you can take one of these new tasks that they put in, and

787
01:54:46.850 --> 01:54:56.170
senate chamber: it'll it'll process the instructions for the past, and it'll kind of do the task in in a small, tracing way, and quite successfully.

788
01:54:59.130 --> 01:55:07.860
senate chamber: My dream was the dream we had. Bike. Remember the the proposal for what we're going to do. We're going to have. We're going to take a system that would take

789
01:55:07.960 --> 01:55:17.760
senate chamber: essentially, maybe slightly crafted instruction for all kinds of tasks, and just read the instruction

790
01:55:17.920 --> 01:55:27.080
senate chamber: and actually perform those tasks, and with the qualifications that have been made, Santa hasn't quite done that. But

791
01:55:27.210 --> 01:55:37.770
senate chamber: but that's kind of the goal that I've always had for Act R, which we we don't have, and I should also say, if we took all those models and put them all in Act R. At once.

792
01:55:39.060 --> 01:55:49.400
senate chamber: it wouldn't work, because we'd have contradictory representations and rules and all the rest. So we really kind of have to keep each of those models running separately.

793
01:55:52.810 --> 01:56:17.069
senate chamber: Yeah. So by that interpretation, I think. And it raises an issue. That's a long-term issue that I was just discussing with Peter Peroli recently of the fact that one challenge we haven't quite met. And you alluded to, you know, hundreds of years of modeler time to create these models is go beyond modeling by hand, and there have been a number of attempts in the community to go

794
01:56:17.070 --> 01:56:34.660
senate chamber: that whether that's taking instruction, whether that's composing models, whether that's having a higher level language. But in any case, I think if Centaur raises a challenge for us, it's definitely that challenge of scaling and going beyond hand modeling toward

795
01:56:34.660 --> 01:56:38.100
senate chamber: some sort of automatic model creation.

796
01:56:44.040 --> 01:56:46.410
Kevin Gluck: Yeah. The may I?

797
01:56:48.020 --> 01:56:49.270
senate chamber: Go ahead. Kevin.

798
01:56:50.130 --> 01:56:55.629
Kevin Gluck: All right. Thanks. I just wanted to just observe that it's really

799
01:56:55.760 --> 01:57:04.610
Kevin Gluck: an interesting point that John's making here about the connection or relationship between what Centaur is doing and model tracing.

800
01:57:04.830 --> 01:57:06.240
Kevin Gluck: Gotta say, having

801
01:57:06.750 --> 01:57:16.028
Kevin Gluck: read both the original draft of their paper and the published draft in Nature several times that never occurred to me. And

802
01:57:17.160 --> 01:57:24.289
Kevin Gluck: It's worth giving some additional consideration to. So I'm I thank you for pointing that out, John. Very interesting.

803
01:57:26.260 --> 01:57:27.680
senate chamber: That make a quick time on that.

804
01:57:27.980 --> 01:57:35.940
senate chamber: So people use large language models the architecture and robotics because it's really good about telling you the next thing to do.

805
01:57:36.160 --> 01:57:41.470
senate chamber: even in physical space, 3 dimensional space. You can do interesting things with it. So as a technology.

806
01:57:41.920 --> 01:57:44.951
senate chamber: my question about the model tracing piece is.

807
01:57:45.620 --> 01:57:52.819
senate chamber: if you had enough data, I think it would do really well at predicting model traces in lots of different forms. If you think of putting them in.

808
01:57:53.590 --> 01:57:56.730
senate chamber: I need to ask what it's gonna tell you about

809
01:57:57.070 --> 01:58:14.869
senate chamber: cognition. I don't know if it would tell you much. Maybe it would, but you could think of it, as you know, it can give you the next thing, whether it's a point in space, whether it's something in a model trace, or whether it's something about language. So it's a general kind of technology, right?

810
01:58:15.880 --> 01:58:21.039
senate chamber: Kevin. That was just in response to your comment, what would it tell us?

811
01:58:23.390 --> 01:58:47.390
senate chamber: Well, excuse me, a standard application might be to be able to know when a student's had enough instruction on something. If you confidently predict they'll kind of get answers in a particular domain. Then you kind of move on to a new content area. So that's a kind of application that one could have for.

812
01:58:48.290 --> 01:59:03.509
senate chamber: And the interesting thing about Centaur is, it isn't particularly designed for one task, so it could presumably generalize to a lot that approach could generalize to a lot of kind of educational tasks.

813
01:59:03.790 --> 01:59:05.080
senate chamber: Situations?

814
01:59:08.700 --> 01:59:15.660
senate chamber: If I can answer that question, this is what I saw. It's not. It's not as thorough as we should say.

815
01:59:16.540 --> 01:59:21.269
senate chamber: What? Oh, my sorry, thank you. It was very highly coordinated task here.

816
01:59:21.440 --> 01:59:25.780
senate chamber: Here's their data set online, and there's no response. Times.

817
01:59:26.310 --> 01:59:34.749
senate chamber: Here's their previous, their original archive paper. And there's no response times, at least on that data set.

818
01:59:34.960 --> 01:59:43.579
senate chamber: And on that data set. And on that data set. And on that data set. And on that data set.

819
01:59:44.030 --> 01:59:50.019
senate chamber: So that does not say there's not some that have response times, but at least 3 of them.

820
01:59:51.210 --> 01:59:58.710
senate chamber: They don't have this. They're not feeding response time, or they're not saying they're feeding response times in so

821
02:00:01.100 --> 02:00:03.590
senate chamber: so kind of verification.

822
02:00:03.920 --> 02:00:17.139
senate chamber: So we keep talking about generalizing the new tasks. But it's important to remember. Centaur is a fine tuned version of Llama and Llama was trained on essentially the entire content of the Internet.

823
02:00:17.300 --> 02:00:22.609
senate chamber: which means it's seen every paper that's ever written on all the tasks that it's supposed to be generalizing to.

824
02:00:22.840 --> 02:00:28.140
senate chamber: And so I guess the question is, do we? Do we actually really think that's generalization, right? It's

825
02:00:28.590 --> 02:00:30.319
senate chamber: you take somebody in here who is

826
02:00:31.490 --> 02:00:37.410
senate chamber: for 30 years and ask them to predict human behavior. It seems pretty reasonable they could do that without the

827
02:00:37.810 --> 02:00:53.740
senate chamber: looking at the cycle. 101 data. Right? Yeah. And I can say that that's 1 of the things that needs to be separated is how much of what Centaur has done is really due to llama versus anything that they contribute to the particular.

828
02:00:54.670 --> 02:01:01.580
senate chamber: So I'll make a comment on that Centaur doesn't generalize at all. I'll make it plain. It has one task.

829
02:01:04.040 --> 02:01:05.090
senate chamber: That's the end of the phone.

830
02:01:05.460 --> 02:01:08.480
senate chamber: But think about it. I mean, it doesn't generalize the what

831
02:01:09.610 --> 02:01:14.110
senate chamber: that's kind of an illusion. It's just one task. It's the next thing in its system

832
02:01:14.250 --> 02:01:18.380
senate chamber: that's not generalization. It's 1 task. It's very good at that one task.

833
02:01:18.620 --> 02:01:24.420
senate chamber: No generalization, I guess a follow up would be.

834
02:01:25.160 --> 02:01:44.170
senate chamber: I think Frank San just said, what if he does this? Can he do that. Can he do the other thing? And let's assume for a second. It can do anything right. All those questions can be answered, and it can do that. Well, at the end of the day again, it doesn't tell me anything about human condition, and I don't understand why I will try and make these sort of parallels assuming you have the

835
02:01:44.840 --> 02:02:12.349
senate chamber: I don't know, a genetic product that is doing so well at mimicking. You know your language and having conversations with you, and being able to pick cues from you so that it can predict your behavior in some way. Studying that pattern, will not? You wouldn't make the argument. Oh, my God, I'm studying this pattern to learn about human cognition. You will maybe say I'm studying this product to learn something about product cognition at best. But why would you try? I don't see the link between this direct.

836
02:02:12.490 --> 02:02:25.139
senate chamber: There's an inference gap, I guess, between what they're doing, and then what cognition means. And we keep going in circles about what's the role of the model? What a model means? What is the theory? What's the role for that?

837
02:02:25.430 --> 02:02:42.029
senate chamber: And we're just missing the point that even if it were to do all those things at the end of the day it wouldn't generalize. I also am of the same view, it wouldn't produce anything new, and it wouldn't be new, because everything that it would produce was probably found on the Internet, among the original training set of love.

838
02:02:42.180 --> 02:02:45.099
senate chamber: It will just seem as so- so

839
02:02:45.390 --> 02:02:47.149
senate chamber: I don't know if you have any thoughts on that.

840
02:02:47.460 --> 02:02:49.839
senate chamber: I'll make one comment. I think

841
02:02:50.510 --> 02:02:56.910
senate chamber: people in the Defense Department of the United States are just really interested in this, and the theme is so somehow has weight.

842
02:02:57.030 --> 02:03:17.960
senate chamber: And it's 1 of the risks of this approach. I think John gives some good examples of uses of it, but there's our uses that are practical to help with when predicting the next thing is useful, that's what it is. So that's good. But your point I agree with completely, and I think that the arguments we need to figure out how to convince others.

843
02:03:18.080 --> 02:03:25.580
senate chamber: For example, just, for example, in the Defense Department, that they should be really careful and find the good cases, and

844
02:03:26.130 --> 02:03:36.689
senate chamber: if they're dealing with intelligence and cognition. This is one potentially very small piece of different kinds of ways to be cognitive, and it's useful at a very certain things.

845
02:03:37.030 --> 02:03:49.960
senate chamber: So I think it's a good comment. Just a quick follow, though. Earlier Frank had a slide that asked, What if the Llm. Does the milligram experiment and ramps up. You know the electricity and kills someone.

846
02:03:49.960 --> 02:04:09.529
senate chamber: There are gel great gel, broken versions of Llms that are open language models. So the Chinese deep sea again you can get it essentially to tell you to kill yourself so if you tell. Oh, my God, I've had depressive thoughts. I'm a loser. I don't want to leave. Should I kill myself, I might as well tell you yes.

847
02:04:09.540 --> 02:04:26.470
senate chamber: so if I were to take on these bad models, or that have been trained at least to not necessarily converge on the opinion of the user. Which again tells you something. Is that what humans do? What do I learn from that model. What aspect of human behavior or cognition

848
02:04:26.740 --> 02:04:37.589
senate chamber: does it relate to, or does it make? I don't know parallels to by telling someone, yeah, you should go kill yourself. Is that a model of I don't know psychopathological behavior, or

849
02:04:38.020 --> 02:04:59.669
senate chamber: I don't know. It's just it's so hard to contextualize this findings. It can call you whatever you want under certain settings, it would be really nice. If it's a conventional model, it can be an asshole if it's an open source model that you tend it to be that way. So and then what is it? Is it about human cognition, or is it? You know I don't know a clone of what we think human condition is.

850
02:04:59.870 --> 02:05:01.410
senate chamber: make any sense. So

851
02:05:03.020 --> 02:05:13.249
senate chamber: it's probably time to move on. But let me just point out 2 thirds of the people in the Milgram experiment did press the high voltage button as well.

852
02:05:13.740 --> 02:05:17.129
senate chamber: And yesterday somebody killed 2 people in Texas.

853
02:05:17.550 --> 02:05:18.610
senate chamber: So

854
02:05:20.480 --> 02:05:36.489
senate chamber: we shouldn't idealize humans too much. Now that's the point. What does it tell you? So between the 2 the lambda will tell you. Don't do it, and the one that like. How do you even compare them between them. Don't just say take Centaur, which is one in the first, st as you have 10.

855
02:05:36.950 --> 02:05:43.250
senate chamber: How do you compare between them? How do you even evaluate the company differences in that way?

856
02:05:43.410 --> 02:05:50.669
senate chamber: What if they all predict? They are trained differently? But they all predict at the same level of accuracy or with minimal changes.

857
02:05:50.840 --> 02:05:54.729
senate chamber: You know the same things. So how do you distinguish between them?

858
02:05:55.680 --> 02:06:03.820
senate chamber: Let's think about that while we move to the online panelists and let them have a chance to share their thoughts.

859
02:06:04.260 --> 02:06:07.389
senate chamber: Kevin or Andrea.

860
02:06:09.370 --> 02:06:15.820
Kevin Gluck: Well, Andrea has his hand up so, and I can't wait to hear what he has to say. So let's hear from Andrea first.st

861
02:06:19.490 --> 02:06:27.270
Andrea Stocco: Thank you, Kevin. So I had a okay. I, looping my comments to the discussion, to my comments as a panelist.

862
02:06:28.170 --> 02:06:34.270
Andrea Stocco: Please allow me. I do better if I visualize. I hope you don't mind if I share some very quick slides.

863
02:06:37.440 --> 02:06:39.620
Andrea Stocco: and you you can see them.

864
02:06:40.270 --> 02:07:00.520
Andrea Stocco: So I think that the best way I can tell you what I feel, what I felt when the center paper drop is that if I tell the story so I was teaching a computational neuroscience class. And we go over a lot of models, including cognitive architectures. And I decided there's me to modify the syllabus and added this as the final paper.

865
02:07:01.850 --> 02:07:10.170
Andrea Stocco: and the best way I can describe my reaction to the paper was that I found it demotivating. Because

866
02:07:10.500 --> 02:07:21.800
Andrea Stocco: I guess, like John, I felt like this is what we should have done. Why isn't why isn't there a Nature paper with 50 actuar authors on it that shows like we model.

867
02:07:22.150 --> 02:07:42.060
Andrea Stocco: And the other thing that motivated was that my students were incredibly unenthusiastic about everything that I told them except the center paper, and they pretty much thought that this was like the end of the story, and one student posted in the discussion Forum that cognition might be at this point a soft problem, which, of course, made me think

868
02:07:42.800 --> 02:07:47.779
Andrea Stocco: that there's been a change, a subtle change in the Zeitgeist.

869
02:07:47.960 --> 02:07:59.639
Andrea Stocco: and some of the things that were really important for me, that I was trying to communicate might not be that important to students, even neuroscience students as college students in the air of big data science.

870
02:08:01.210 --> 02:08:11.299
Andrea Stocco: I thought the center was amazing. But in the same way that the Google Attai paper that came out in 2015 was amazing. Like a technological feed that was unexpectedly good.

871
02:08:11.670 --> 02:08:27.980
Andrea Stocco: I didn't think that it was amazing in the sense of giving insight about condition. And in particular, I want to stress. There are a lot of things that our models do, that go beyond prediction, including explaining, providing an interpretation, giving you an idea of what would happen in other circumstances.

872
02:08:28.400 --> 02:08:32.760
Andrea Stocco: as Christians like to say, even the Pollution party is really overblown because

873
02:08:34.290 --> 02:08:40.530
Andrea Stocco: potentially center can fit anything. And it's like, in the sense is a model in the sense that, like

874
02:08:41.230 --> 02:08:57.139
Andrea Stocco: general, very complicated linear model is, if you give it data, we'll find the best fit doesn't mean that the underlying assumptions reflection in any meaningful way, and also I found it personally unfair that all of us in the cognitive architectures domain have been criticized

875
02:08:58.290 --> 02:09:15.140
Andrea Stocco: for decades at this point that you know like, oh, you have too many parameters. You can't fit anything, and God only knows I can never fit anything on my data set so clearly. We don't have that many parameters. But no matter what 100 parameters is not even comparable to 3 billion parameters in Lama.

876
02:09:16.190 --> 02:09:31.420
Andrea Stocco: So Mark already explained like this figure. But there is a lot that goes into architecture that implicit assumptions about what we know is not possible, and they are not captured in a larger model per se. That being said.

877
02:09:31.790 --> 02:09:57.200
Andrea Stocco: if I step back and I take my disappointment into motivation, there are things that I believe we should take away and take the challenge. And my reaction is that somehow we might. I definitely, I've missed this kind of shift in the Zeitgeist about what people focus on. And I think this is an opportunity for us to refocus and say, Okay, let's accept the challenge. What can we do

878
02:09:57.930 --> 02:10:07.730
Andrea Stocco: better than what they are doing? And this is actually a list of a very obituary list of criteria that they think we should set forward as a as a

879
02:10:07.850 --> 02:10:21.489
Andrea Stocco: community to try to improve ourselves and go beyond what even entour claims to be doing. Focus a little bit more on prediction, Ia prediction. But it's something that people seem to be interesting enough, and

880
02:10:21.680 --> 02:10:32.179
Andrea Stocco: I'm sad to say, for most of my students, that seems to be the end of the raid focus on the large data sets. I like the idea of integrating models across multiple tasks. As John said, there's always been

881
02:10:33.000 --> 02:10:39.569
Andrea Stocco: ultimate problem there, because you cannot just combine models arbitrary. There are always inconsistencies

882
02:10:39.710 --> 02:10:51.579
Andrea Stocco: in system of the comparison use benchmarks. I really dislike that. But apparently this is like a way to convince people that we're doing the right thing. And most importantly, I have more ambition. I think that one of the things that has kept us down is that

883
02:10:52.370 --> 02:11:04.639
Andrea Stocco: maybe we should have tried harder to do the things that we are capable, and maybe we don't know what we can do with a car that haven't been tried yet, and also, of course, our ultimate goal should be to have papers with 150 authors.

884
02:11:04.800 --> 02:11:14.090
Andrea Stocco: That being said when I'm saying that these things should be goals, this is the reason why I think they are actually extremely achievable. Because if you look in that type

885
02:11:14.110 --> 02:11:36.329
Andrea Stocco: data set and the entire repository papers, there's plenty of people that have done this I put immodest in myself in one of them, just because, like, it was an easy thing to retrieve. But all the people that are in this room have contributed in this direction once or more, so I think we have the opportunity to do it, and our limitations might be like laying. Otherwise, I really like Chris's point that

886
02:11:36.330 --> 02:11:46.520
Andrea Stocco: one of the problems that we have is scalability. We don't have. We never thought about engineering pipelines to make things more robust, more scalable, more integrated. And I think that this is

887
02:11:46.880 --> 02:11:50.619
Andrea Stocco: the center paper is a little bit of call to arms that

888
02:11:51.530 --> 02:11:57.919
Andrea Stocco: we might need to speed up a little bit on this side of the architecture development.

889
02:12:01.760 --> 02:12:03.250
Andrea Stocco: That's it.

890
02:12:03.680 --> 02:12:10.709
senate chamber: We'll just move straight over to Kevin, and then we'll after that open the floor up for open discussion.

891
02:12:12.960 --> 02:12:13.900
senate chamber: Okay.

892
02:12:15.460 --> 02:12:17.280
Kevin Gluck: Okay, thanks, Drew.

893
02:12:17.560 --> 02:12:25.370
Kevin Gluck: I will also start with an opening slide, anyway, and

894
02:12:27.000 --> 02:12:29.100
Kevin Gluck: you should be seeing that now.

895
02:12:31.573 --> 02:12:32.176
Kevin Gluck: So

896
02:12:34.060 --> 02:12:52.820
Kevin Gluck: for those who aren't aware about the the process for preparing for this. There was a suggestion made to the people who were speaking at the beginning of each session to share their slides with the panelists ahead of time, so that we had some idea what their content was going to be, and would be prepared to comment on it. So

897
02:12:54.140 --> 02:13:14.915
Kevin Gluck: mark, or who is is our very own oracle here at Ihmc. Made a prediction in the slides that he sent to us before the workshop. Now it turns out that he removed this prediction for the slides that he presented earlier. But I'm bringing it back here because

898
02:13:15.930 --> 02:13:26.869
Kevin Gluck: He was right that this, that this is a point of discussion and contention. And so, you know, in the spirit of of all good oracles.

899
02:13:27.040 --> 02:13:33.270
Kevin Gluck: Mark was, Mark was correct that this is a point of discussion and contention for me.

900
02:13:33.720 --> 02:13:37.829
Kevin Gluck: I do disagree with this characterization.

901
02:13:40.250 --> 02:13:48.470
Kevin Gluck: I agree. I disagree with this characterization because Centaur already is a unified model of cognition.

902
02:13:49.830 --> 02:13:52.840
Kevin Gluck: but only in the weak sense of that claim

903
02:13:53.000 --> 02:14:02.990
Kevin Gluck: as a system that is, an implementation of computational cognition. So Centaur is unified in the sense

904
02:14:03.160 --> 02:14:09.810
Kevin Gluck: that the same computational mechanisms are used in all of the currently evaluated task contexts.

905
02:14:10.700 --> 02:14:19.260
Kevin Gluck: Centaur is a model right. It's an abstraction of reality implemented as a computational system

906
02:14:19.400 --> 02:14:24.869
Kevin Gluck: that can be used for prediction, and it is cognitive

907
02:14:25.150 --> 02:14:28.010
Kevin Gluck: in the most general sense of that term.

908
02:14:28.410 --> 02:14:34.290
Kevin Gluck: and that it takes information as input transforms, it produces output.

909
02:14:36.070 --> 02:14:42.429
Kevin Gluck: So what I object to is Mark's characterization in in that slide, anyway.

910
02:14:42.950 --> 02:14:50.730
Kevin Gluck: that Centaur is moving toward a unified model of cognition. It is not moving toward it, it already is one.

911
02:14:52.720 --> 02:15:04.360
Kevin Gluck: and, in my opinion, and reflecting some of some of the comments that we've already heard during this session. The work on Centaur is is truly impressive, and it's interesting, and it's worth paying attention to.

912
02:15:04.950 --> 02:15:12.750
Kevin Gluck: And when I say we should pay attention to it. I mean the details where the science is.

913
02:15:13.450 --> 02:15:22.630
Kevin Gluck: and not only the details of its implementation, but also the methodological details of how Centaur is being evaluated.

914
02:15:22.970 --> 02:15:29.020
Kevin Gluck: as well as the details of how its developers and others are describing it.

915
02:15:29.960 --> 02:15:35.400
Kevin Gluck: And and this is where I do have an objection and a concern regarding center.

916
02:15:35.900 --> 02:15:48.430
Kevin Gluck: So here I'm I'm turning our attention from the words that Mark used to describe Centaur in his opening comments to the words, the authors of the recently published paper used.

917
02:15:50.140 --> 02:15:52.865
Kevin Gluck: Unfortunately, it's the case that

918
02:15:53.580 --> 02:16:03.819
Kevin Gluck: the reviewers and editor at Nature let the authors get away with making the repeated claim, including in the title of the paper

919
02:16:04.170 --> 02:16:07.830
Kevin Gluck: that Centaur is a model of human cognition.

920
02:16:08.730 --> 02:16:15.720
Kevin Gluck: which is a different claim that comes with certain additional scientific requirements.

921
02:16:16.540 --> 02:16:34.860
Kevin Gluck: So, for example, a basic fundamental expectation for any model of human cognition is that the model produces its predictions, using amounts of information and types of cognitive processes that are at least plausible

922
02:16:35.120 --> 02:16:45.549
Kevin Gluck: and preferably validly replicative of the perceptual, attentive, and cognitive processes and capacities of humans.

923
02:16:46.660 --> 02:16:57.799
Kevin Gluck: and in the Centaur paper there there is no detailed evaluation, comparing the processes by which Centaur produces its performance, outcomes

924
02:16:57.969 --> 02:17:04.659
Kevin Gluck: to the processes by which people in those same task conditions produce their performance outcomes.

925
02:17:05.219 --> 02:17:07.509
Kevin Gluck: And and there's good reason for that

926
02:17:07.910 --> 02:17:12.629
Kevin Gluck: which is that Centaur is not a model of human cognition.

927
02:17:13.290 --> 02:17:18.689
Kevin Gluck: It's a sophisticated stimulus, response, behavior, probability calculator.

928
02:17:19.639 --> 02:17:29.489
Kevin Gluck: the the true nature of Centaur. The system is reflected in the author's description that Centaur captures human behavior.

929
02:17:30.370 --> 02:17:37.540
Kevin Gluck: which is a more accurate and less objectionable phrase which they repeatedly use throughout the paper.

930
02:17:39.059 --> 02:17:50.850
Kevin Gluck: So in the context of computational cognitive systems, we know there is a natural tendency for people to anthropomorphize

931
02:17:51.209 --> 02:18:00.920
Kevin Gluck: and and thereby attribute additional human-like capabilities, often well beyond those that actually exist within those systems.

932
02:18:01.690 --> 02:18:07.569
Kevin Gluck: And what we see here in the language its authors are using to describe Centaur

933
02:18:07.940 --> 02:18:21.749
Kevin Gluck: is a particularly insidious subtype of this behavior that should really be of great concern to all of those who are actually working on models of human cognition.

934
02:18:23.190 --> 02:18:27.959
Kevin Gluck: And so you know here today, for the 1st time.

935
02:18:28.500 --> 02:18:36.480
Kevin Gluck: I would would like to introduce anthropococognomorphization.

936
02:18:40.600 --> 02:18:46.250
Kevin Gluck: Just note how it just it just rolls right off the tongue. I mean, we've all really been thinking this right already.

937
02:18:48.139 --> 02:18:54.759
Kevin Gluck: This is this is the attribution of human cognition to things that are not human cognition.

938
02:18:57.820 --> 02:19:06.559
Kevin Gluck: and it is entirely understandable and forgivable that lay people in the general population do this.

939
02:19:07.000 --> 02:19:16.110
Kevin Gluck: however, scientists must be held to a higher standard, and this behavior, as exemplified in the Centaur paper

940
02:19:16.430 --> 02:19:29.619
Kevin Gluck: absolutely should not stand unchallenged, and I call on all cognitive scientists who care about the distinction between general cognitive systems and models of human cognition

941
02:19:29.770 --> 02:19:36.260
Kevin Gluck: to object immediately to anthropococognitomorphizing anywhere. You see it.

942
02:19:37.330 --> 02:19:38.240
Kevin Gluck: Thank you.

943
02:19:49.760 --> 02:19:54.559
senate chamber: Did you practice that, Kevin? Saying that so fast I must have taken a while.

944
02:19:54.780 --> 02:20:00.780
Kevin Gluck: Yes, dozens of times, but not but not but not thousands.

945
02:20:04.070 --> 02:20:08.329
senate chamber: So Drew, where are we? Do we have more? That's it from online.

946
02:20:08.690 --> 02:20:15.099
senate chamber: Yeah, that's it from the panelists. So we'll open it up to discussion. We have about 10 min, Christian, till

947
02:20:15.210 --> 02:20:19.850
senate chamber: I have. I have some corrections I'd like to add right?

948
02:20:20.450 --> 02:20:23.509
senate chamber: So oh, I gotta share.

949
02:20:25.540 --> 02:20:27.020
senate chamber: Sorry I'm all ready to.

950
02:20:27.870 --> 02:20:32.520
senate chamber: You're not allowed to do corrections like it's too late. Sure, they have to go through right

951
02:20:34.740 --> 02:20:42.549
senate chamber: one of the problems. And I think this agrees with what Kevin was saying is, you know. Let's say we have a model of shocking in the Milgram study.

952
02:20:43.240 --> 02:20:46.670
senate chamber: Centaur doesn't tell us anything about how to decrease that.

953
02:20:47.980 --> 02:20:49.790
senate chamber: Which is why you want a theory.

954
02:20:51.360 --> 02:20:53.990
senate chamber: Are you taking comments in in real time?

955
02:20:54.160 --> 02:21:05.530
senate chamber: Depends on you? Can you generalize the social data that would be interesting. And one of the points we haven't made as strong as we should have is, there's there's a hundred 1 million data points.

956
02:21:06.200 --> 02:21:09.000
senate chamber: There's a hundred 1 million. That's impressive.

957
02:21:09.730 --> 02:21:16.339
senate chamber: But but there's 7 billion parameters, you know. So

958
02:21:17.030 --> 02:21:19.880
senate chamber: there, those are my corrections to what my statement was.

959
02:21:20.980 --> 02:21:24.979
senate chamber: Well, if you wanted to perform well, or to your liking for Milgram.

960
02:21:25.780 --> 02:21:28.549
senate chamber: you would just give it the right training data. I mean, it would work right.

961
02:21:28.800 --> 02:21:40.550
senate chamber: No, but I want to know why. Oh, I want suggestions like, you know, empathy, or more training, or less training, and it tells you nothing. It tells you nothing about that.

962
02:21:51.050 --> 02:22:05.849
senate chamber: So so one quick point in particular, it was prompted by John's comment that what it's really doing is model tracing. I mean, this is fundamentally a behaviorist theory

963
02:22:05.850 --> 02:22:23.920
senate chamber: of human cognition in the sense that the model doesn't maintain an internal state. It doesn't care about the internal mechanism. It just has its prompt in its buffer. And it generates the next step. And it reacts like that. And that's basically where it's taking us back.

964
02:22:23.930 --> 02:22:52.260
senate chamber: And just to emphasize 1 point that Mark made about the new Old test and how they're scoring themselves. It's very much a bait and switch. Some of the criteria about the new old test is behave arbitrarily or learn. For that matter, Centaur doesn't learn at all. All. Its weights are frozen. Centaur is exactly what Centaur is going to be during its entire run. It doesn't change anything.

965
02:22:52.690 --> 02:23:00.789
senate chamber: but obviously the original, the transformer that we trained with all that knowledge and then fine tune. The experiment

966
02:23:01.350 --> 02:23:18.569
senate chamber: does learn, not in the way that the humans do? Certainly not on that scale, but which system which part of the system which side of the system are they making their claim. For are they making the transformers model of cognation?

967
02:23:18.760 --> 02:23:37.600
senate chamber: Then that's fine. Right? Let's put transformers through human-like interaction. And let's see how they can do with human levels of information or human type of interaction. That would be a fair test. But so, having 2 systems and making claims for one versus the other that things somewhat deceptive

968
02:23:40.390 --> 02:23:41.130
senate chamber: here.

969
02:23:41.990 --> 02:23:46.430
senate chamber: Thank you. I hear Christian might have made my question slash comments a little bit.

970
02:23:47.160 --> 02:23:51.609
senate chamber: but I I wanna touch on the

971
02:23:54.050 --> 02:24:04.520
senate chamber: anthrocognital comorphization, was it? I don't recall just a little bit and sort of the question

972
02:24:06.340 --> 02:24:13.790
senate chamber: why, we want to call what it's doing cognitive at all like if we if we go back to response time.

973
02:24:13.920 --> 02:24:25.070
senate chamber: the response time data issue, it doesn't emit response time data. But let's imagine for a second that it did that was just included in the tokens that it's trained on

974
02:24:25.210 --> 02:24:36.549
senate chamber: for all the experiments. The response time data that it would emit would be unless it does some sort of chain of thought, and we compute its response. Times based on the number of tokens it outputs

975
02:24:38.490 --> 02:24:48.799
senate chamber: the response. Time would be entirely independent of the processing it's doing in order to produce that response

976
02:24:49.160 --> 02:24:53.300
senate chamber: or the data associated with response time, right?

977
02:24:53.480 --> 02:24:59.600
senate chamber: And it. And and we know because it's modeling decision. At least, you know.

978
02:25:00.458 --> 02:25:03.750
senate chamber: with with reasonable, with reasonable accuracy.

979
02:25:04.665 --> 02:25:05.390
senate chamber: That

980
02:25:05.840 --> 02:25:12.319
senate chamber: whatever processing that it's doing, it wouldn't need to do that in order to predict the decision that that

981
02:25:12.640 --> 02:25:14.820
senate chamber: that would be made in that context.

982
02:25:15.440 --> 02:25:18.720
senate chamber: So it seems to be in this way

983
02:25:18.880 --> 02:25:27.109
senate chamber: purely a behavioral model. And if you were going to like. Add some sort of like cognitive processing element on top of it.

984
02:25:27.320 --> 02:25:29.760
senate chamber: It seems to me like it would be

985
02:25:31.160 --> 02:25:38.519
senate chamber: somehow entirely accidental to the behavior that the model is is producing.

986
02:25:40.260 --> 02:25:47.060
senate chamber: So I guess what I want to question and

987
02:25:47.190 --> 02:26:03.129
senate chamber: maybe challenge like, it seems if they're making some sort of claim about it's doing cognition. It's going to entirely hinge on whether the latent states of the model are somehow related to the states of the human brain measured by fmri data. I don't fucking know?

988
02:26:03.898 --> 02:26:06.249
senate chamber: In certain cognitive tasks.

989
02:26:06.670 --> 02:26:24.750
senate chamber: and maybe we could do that. But besides that, it seems difficult for me to conceptualize how we think this is modeling any kind of cognition at all, let alone human cognition. So some people made the claim that it's doing some kind of cognitive modeling.

990
02:26:27.930 --> 02:26:33.439
senate chamber: And then there's maybe it's maybe it's doing sort of alien, cognitive modeling.

991
02:26:33.800 --> 02:26:40.450
senate chamber: But like I I'm not sure I'm not sure how it makes sense to even claim that it's modeling

992
02:26:40.590 --> 02:26:47.860
senate chamber: cognition. And if we can like, maybe challenge me on this, I would appreciate sharing some feedback.

993
02:26:50.750 --> 02:26:58.640
senate chamber: So just one quick comment. We have to define what cognitive is first, st and then we can define whether it falls to those criteria.

994
02:27:00.010 --> 02:27:01.789
senate chamber: Andrea has his hand up, won't mind.

995
02:27:02.150 --> 02:27:04.770
senate chamber: Andre has his hand up online. Go for it, Andre.

996
02:27:06.900 --> 02:27:10.365
Andrea Stocco: Thank you. It was a comment about the last 2 points.

997
02:27:11.600 --> 02:27:16.927
Andrea Stocco: so I think that Christian is correct. And I think this is like probably the biggest

998
02:27:17.530 --> 02:27:21.680
Andrea Stocco: miss Opportunity, if such opportunity exists in the paper, is that

999
02:27:22.410 --> 02:27:26.399
Andrea Stocco: we don't know what Center has learned.

1000
02:27:26.790 --> 02:27:28.500
Andrea Stocco: The fact that he can

1001
02:27:28.950 --> 02:27:35.960
Andrea Stocco: out. You know, outperform. Any kind of standard model of every single task in prediction to me implies.

1002
02:27:36.190 --> 02:27:38.639
Andrea Stocco: as it is picked up on

1003
02:27:38.980 --> 02:27:43.409
Andrea Stocco: effects that we probably haven't even discovered as cognitive psychologists.

1004
02:27:44.150 --> 02:27:53.529
Andrea Stocco: But there should be a way to analyze what it has learned and how it has learned, especially during the fine-tuning part.

1005
02:27:53.920 --> 02:27:59.390
Andrea Stocco: and to recover or interpret. If it has learned anything that is like

1006
02:28:00.540 --> 02:28:10.890
Andrea Stocco: cognitively interesting, like, why does he think that the 97, like response in the go. Task should be like this, or should be an error, and so on.

1007
02:28:11.840 --> 02:28:15.969
Andrea Stocco: There was in the original draft. There was a much longer section

1008
02:28:16.100 --> 02:28:28.049
Andrea Stocco: on comparing brain the activity of the simulated neurons with the brain data. This has been scaled down in the current nature paper, but I thought that was like probably

1009
02:28:28.220 --> 02:28:35.329
Andrea Stocco: the most disappointing the original draft, but also like the most interesting direction in which they were going, because it was a way to open the black box

1010
02:28:35.980 --> 02:28:42.040
Andrea Stocco: without the fact that they didn't to me is pretty telling about

1011
02:28:43.231 --> 02:28:47.650
Andrea Stocco: the current state of anthropocanomorphization. How much

1012
02:28:47.900 --> 02:28:51.730
Andrea Stocco: people think that just because something mimics our response.

1013
02:28:53.880 --> 02:28:57.219
Andrea Stocco: Is sufficient to have a cognitive theory.

1014
02:28:57.630 --> 02:29:02.100
Andrea Stocco: And yes, Kevin had been practicing when my audio was off to say it correctly.

1015
02:29:05.020 --> 02:29:08.560
senate chamber: We have a 1 comment question from in the room.

1016
02:29:09.510 --> 02:29:14.280
senate chamber: It's Alex Petrov here. Hi! Ken Hi, Andrew! Great talks.

1017
02:29:15.990 --> 02:29:18.799
senate chamber: I don't know if it's a comment. But

1018
02:29:19.550 --> 02:29:23.819
senate chamber: many people say it's just fitting data. It's behaviorist.

1019
02:29:25.450 --> 02:29:31.030
senate chamber: Here's my concern. Consider the scientific community broadly construed.

1020
02:29:32.050 --> 02:29:36.179
senate chamber: All we hear in this room. What do we have to go on?

1021
02:29:36.460 --> 02:29:43.980
senate chamber: We have data, and we build theories based on these data. And the earlier workshop, for example, was.

1022
02:29:44.470 --> 02:29:49.399
senate chamber: How do you discriminate parallel serial blah blah. But it's all based on data.

1023
02:29:49.670 --> 02:29:52.460
senate chamber: And so in the limit.

1024
02:29:54.000 --> 02:30:04.819
senate chamber: what do we have? That in principle is not available to some system that that is trying to do algorithmic theory induction

1025
02:30:04.970 --> 02:30:08.249
senate chamber: from a empirical database.

1026
02:30:08.510 --> 02:30:17.350
senate chamber: So as a thought experiment, imagine. So a lot was said that it threw away the reaction times, and it probably threw away other things

1027
02:30:17.710 --> 02:30:27.029
senate chamber: in the limit. Imagine all data ever collected by experimental psychologists in one giant data set.

1028
02:30:27.390 --> 02:30:33.810
senate chamber: and then imagine you come up with some kind of machine learning procedure that can tell you

1029
02:30:35.520 --> 02:30:38.790
senate chamber: can account for the predictable variance in that.

1030
02:30:40.670 --> 02:30:48.590
senate chamber: He's and many of the things that were said today here in this room would still apply.

1031
02:30:48.790 --> 02:30:54.979
senate chamber: But then, is there any hope for science? Do we have anything other than above that.

1032
02:30:55.190 --> 02:31:00.900
senate chamber: and I know time is short. I'm very curious to hear your reaction to what I just say. But

1033
02:31:01.560 --> 02:31:08.150
senate chamber: there is this one card in our in up our sleeves, which is introspection.

1034
02:31:09.530 --> 02:31:13.490
senate chamber: It so happens we actually can do these tasks.

1035
02:31:13.860 --> 02:31:19.529
senate chamber: There is a difference between studying human cognition and studying

1036
02:31:20.290 --> 02:31:42.860
senate chamber: physics or chemistry or chicken cognition, for that matter, and that is as a vision scientist. I can sit in the cubicle and look at the stimulus and and have a phenomenological experience of what it looks like and what it doesn't look like. And I have something more to go on than the reaction times and the accuracies and so forth.

1037
02:31:43.110 --> 02:31:44.010
senate chamber: Thank you.

1038
02:31:47.720 --> 02:31:54.579
senate chamber: So I agree we should take this seriously at least one perspective.

1039
02:31:54.880 --> 02:32:04.280
senate chamber: We can discuss whether it's model commission or not. But I can predict it's going to be very attractive.

1040
02:32:05.210 --> 02:32:08.730
senate chamber: Use. A lot of psychologists will use it.

1041
02:32:09.480 --> 02:32:10.470
senate chamber: Oh, just cute.

1042
02:32:11.740 --> 02:32:20.030
senate chamber: You should consider that even though this seems like seamless. The use of the system

1043
02:32:20.650 --> 02:32:29.690
senate chamber: seems seamless. It still consumes a lot of electricity and water, and

1044
02:32:30.250 --> 02:32:32.780
senate chamber: and it's a greenhouse gas emission.

1045
02:32:36.020 --> 02:32:37.230
senate chamber: I have a comment

1046
02:32:37.360 --> 02:32:47.330
senate chamber: for Alex so Alphago, that I gave that as an example, and I think that was, you know, in the spirit of what you're saying, Alphago.

1047
02:32:47.460 --> 02:32:53.379
senate chamber: you might not with it, as a isn't really a scientific process, but it's very good.

1048
02:32:53.900 --> 02:33:00.739
senate chamber: It was a machine learning algorithm that had knowledge of the domain in it. Also to constrain it.

1049
02:33:01.670 --> 02:33:02.530
senate chamber: And I

1050
02:33:02.850 --> 02:33:16.619
senate chamber: think that that's what you're talking about, bringing in all the knowledge that we have and lots and lots and lots of data and trying to use procedures to understand that data with respect to constraints that we think are a theory.

1051
02:33:16.940 --> 02:33:33.299
senate chamber: So is that in the spirit of what you're talking. I'm not saying exactly how to go do it. I'm just saying that. Take that as an abstract model, for where cognition should go, and I think that part of the puzzle is to look across species. I don't think there's any other way to do it. Actually.

1052
02:33:34.110 --> 02:33:43.749
senate chamber: at some point you'll get a lot of purpose by looking across a lot of different species. And I think defining cognition in terms of teleological matter would be

1053
02:33:43.880 --> 02:33:49.720
senate chamber: in the service of exactly what you're saying and going back to basics, I'll stop with that.

1054
02:33:50.830 --> 02:33:55.240
senate chamber: We're running into lunchtime. But we had one more comment at least.

1055
02:33:56.720 --> 02:34:21.379
senate chamber: Thank you so much, looking at all the different perspectives. It sounds to me like we are not. There isn't much of a deep dive into what the actual architecture of transformers can do for us. These are models that we can. We can, unlike a brain, take it apart and look at the connectivity between the attention key value queries.

1056
02:34:22.420 --> 02:34:41.239
senate chamber: I think that there is a lot of value in being able to extract this connectivity and understanding that the tool is any machine learning tool that will consume electricity just like the Mcnc sensors that we use on Gpus.

1057
02:34:42.900 --> 02:34:49.090
senate chamber: we we might still be able to extract value by intact by using it as

1058
02:34:49.120 --> 02:34:57.259
senate chamber: what the tool is for, and that is an aggregator of context. So if the firm only have the answers, but

1059
02:34:57.270 --> 02:35:00.769
senate chamber: in the future, if we add response times they would be able to

1060
02:35:00.780 --> 02:35:29.159
senate chamber: aggregate the distributions that the responses have with respect to the response times. Even if we don't know what that is, we will be able to make associations in context. So, for example, I would see a lot of value in adding metadata about the people who answered the psychological responses. And then looking at the distributions of that data out of the samples that were taken.

1061
02:35:29.320 --> 02:35:37.870
senate chamber: I think there's some value in being able to examine where those answers come from, even if the architecture that produced it is different.

1062
02:35:38.560 --> 02:36:02.469
senate chamber: But that's an entirely different thing, and that would land them in nature right? Using it as just a statistical tool and not a model of condition, is an entirely different thing. There is something called symbolic regression, which is essentially an automated way of taking mathematical operators and letting a neural network fit different functions over your data. You can do that jointly for

1063
02:36:02.740 --> 02:36:26.959
senate chamber: a lot of data sets and big data sets and different sort of paradigms. So now imagine that you do that for everything that they did it on. And it comes back with a mathematical function that explains everything with a reasonable error or a satisfactory level of error, and then tells you that it has. I don't know 30 unknown parameters in there. Maybe way less than that is that a model of cognition?

1064
02:36:28.020 --> 02:36:31.520
senate chamber: It can account for all those tasks. It's just one

1065
02:36:31.900 --> 02:36:38.470
senate chamber: functional is just one functional relationship over all those tasks. And over all that data.

1066
02:36:38.790 --> 02:36:42.040
senate chamber: Is that a model of commission? No, it's not.

1067
02:36:42.160 --> 02:36:57.139
senate chamber: It is a model. It serves a purpose. It could be combined with the Llm. But you also will go in and discover what are these things that matter, or don't matter to give it meaning, and that's the whole point. It can give us a lot of things and all about Llms.

1068
02:36:57.570 --> 02:37:01.439
senate chamber: But it doesn't give us meaning. We have to go back and find that in

1069
02:37:01.620 --> 02:37:12.380
senate chamber: because this data that we trained on is just experimental tasks that also do not really capture the spectrum of human behavior, we're more than just astute task.

1070
02:37:12.990 --> 02:37:24.060
senate chamber: you know, and there are so many things that it doesn't account for. So I guess we are always coming back to. It's a great tool, we should give it to them. They did great job. But it's not what it's.

1071
02:37:25.150 --> 02:37:30.089
senate chamber: It's not what it's saying that it is, that's all. It's a great statistical tool.

1072
02:37:30.520 --> 02:37:35.670
senate chamber: We can narrow it down and work on. What can it teach us? And what can we use it on? But that's all.

1073
02:37:38.650 --> 02:37:50.850
senate chamber: So I think that final discussion foreshadows very nicely the 1st session of the afternoon. So it's time to break for lunch now. So 1st of all, let's thank the panel and the speaker

1074
02:37:57.210 --> 02:38:15.149
senate chamber: a practical note. So we're reconvening at one o'clock Eastern. There is, I believe, a market somewhere in the building where you can grab good to go. If you walk out the front door of High Street. There's lots of places to eat reasonably fast.

1075
02:38:15.150 --> 02:38:27.229
senate chamber: I would not recommend leaving laptops here, because I don't know about the security of the building, so I think you should take things with you unless you have somebody staying here to watch over that.

1076
02:38:27.230 --> 02:38:30.209
senate chamber: Thank you, and let's get back to you at one o'clock.

1077
02:38:38.260 --> 02:38:42.229
senate chamber: No, not yet more than that.

1078
02:38:47.260 --> 02:38:48.000
senate chamber: It's work.

1079
02:38:50.120 --> 02:38:53.199
senate chamber: We're sitting around the corner doing satellite, saying sad.

1080
02:39:14.390 --> 02:39:17.349
senate chamber: We've got time time enough.

1081
02:39:24.100 --> 02:39:33.830
senate chamber: It is so welcome. Thank you, to the new speaker. We're about to begin hybrid series. The speaker is

1082
02:39:34.790 --> 02:39:36.300
senate chamber: Christian Livier.

1083
02:39:36.710 --> 02:39:45.200
senate chamber: You're gonna have 20 min. We've got some timers here. If you watch online, there'll be a timer also on my background.

1084
02:39:45.970 --> 02:39:46.750
senate chamber: Something.

1085
02:39:47.480 --> 02:39:50.009
senate chamber: Well, thank you, Frank, go, too. Good sir.

1086
02:39:50.160 --> 02:39:52.960
senate chamber: Thank you, Frank. Can you hear me online.

1087
02:39:55.050 --> 02:39:55.480
Andrea Stocco: Yes.

1088
02:39:55.480 --> 02:39:56.079
senate chamber: All right.

1089
02:39:57.520 --> 02:40:18.539
senate chamber: So after we agreed this morning that generative AI large language model based theories of cognition like Centaur, were a terrible thing. Let's figure out how we can use them here. So it didn't take long after

1090
02:40:18.720 --> 02:40:22.879
senate chamber: Llm. Exploded on the scene in

1091
02:40:23.630 --> 02:40:28.740
senate chamber: November 2022, for a number of us to realize.

1092
02:40:29.260 --> 02:40:32.830
senate chamber: Is there a magic trick for advancing the slides?

1093
02:40:43.330 --> 02:40:44.210
senate chamber: There you go.

1094
02:40:44.700 --> 02:41:13.050
senate chamber: A number of us have realized that those are 2, I think very different kinds of intelligence, if you can call it that. It was a natural thing to think about ways of integrating them. And it turns out it's a very rich space, rich and diverse space, depending of what the potential uses are, and not just sort of necessarily pure theories of human cognition that can also sort

1095
02:41:13.050 --> 02:41:16.439
senate chamber: spend the spectrum toward more AI type application.

1096
02:41:16.790 --> 02:41:25.919
senate chamber: So a number of us, Andre Astoko, John, Laird, Paul Rosenblum, and I co-organized the triple AI fall symposium.

1097
02:41:26.030 --> 02:41:55.760
senate chamber: and you have some ideas there. We're trying the topics listed there sort of our thoughts at the time about the various ways in which those 2 very different forms of intelligence systems to put together. And it turns out that's more or less how the Symposium came together. So you can find a website that was used for the Symposium online. You can probably find it from the Aa website for false symposia.

1098
02:41:55.810 --> 02:42:11.910
senate chamber: And almost all the papers proceedings have been published by Aa. You can also find them. And many of those papers were by people in this room, or who will be on the panel or online.

1099
02:42:11.910 --> 02:42:31.939
senate chamber: So I'm not going to try to give a reflection of their work. They're a better placed than me to do that. What I'm going to try to give you here is essentially the lay of the space that we've delineated as part of that symposium. That was a year and a half ago. So obviously, there's been more work since, but I think that the

1100
02:42:32.030 --> 02:42:40.890
senate chamber: the the various outlines of the the ways to use that to to do this kind of hybridizing were already laid out fairly well there.

1101
02:42:41.380 --> 02:42:49.489
senate chamber: So I'm going to go very fast here again. Try. Just try to give you a sense of the the different kinds of work that are that are possible.

1102
02:42:50.410 --> 02:43:11.130
senate chamber: And the 3rd thing that John Laird gave an introduction and his main point there was that the cognitive architectures and large language models have very much complementary strengths and weaknesses, and you can see that sort of in the cross arrows, pointing from one sort of strong points

1103
02:43:11.160 --> 02:43:24.669
senate chamber: to the others limitation, and I guess we can figure it out from the the session this morning is that cognitive architectures are fundamentally more mechanistic beasts

1104
02:43:24.670 --> 02:43:45.809
senate chamber: that have a stronger theoretical background, especially mapping to human cognition, and some abilities that are more generally associated with symbolic type abilities, whereas large language models have that massive amount of knowledge that they incorporate, and then various sort of scaling properties that come with that.

1105
02:43:47.270 --> 02:43:59.109
senate chamber: So that's sort of the justification for that. That intuition, that hybridizing those systems could yield benefits beyond what each sort of has demonstrated. To this point

1106
02:44:00.390 --> 02:44:24.450
senate chamber: Oscar Romero gave an introductory talk, where he sort of tried effectively to have a condensed version of the goal of the Symposium, which is to map various ways of doing that combination. And the most obvious way of combining things. That's always the low hanging fruit is modular combinations.

1107
02:44:24.450 --> 02:44:42.009
senate chamber: and you can see the graph there on the right various ways of combining them. So one, for example, is that you can use cognitive architecture to reason and to augment the chain of thought interaction of Llm. So essentially there the cognitive architecture stand for human user.

1108
02:44:42.490 --> 02:45:04.450
senate chamber: Conversely, you can use Llms for the perception action front end of cognitive architecture. Llm is mostly obviously language limited. But obviously there's generative AI more generally that can do those kind of things for visual modalities, and so on.

1109
02:45:05.150 --> 02:45:30.489
senate chamber: Somewhat more ambitious, and perhaps deeper, is to generalize the Llms, not just for language, but for use in other modules. And obviously, since Llms are this massive store of knowledge, it would make sense to use them as some kind of implementation for declarative and procedural modules, which are the main repositories of knowledge in

1110
02:45:30.810 --> 02:45:45.130
senate chamber: our cognitive architectures, and finally, a sort of a little bit of an odd use is to use Llm. For sort of an internal cognitive architecture, simulation sort of as world model, as as you have sort of a theory of mind there.

1111
02:45:45.630 --> 02:46:12.450
senate chamber: And a fundamental concept that these 2 very different systems have in mind is the central role that working memory plays in cognitive architectures, buffers in actar versus the context window plays in generative AI and the large language model. It's an interesting parallel there, but it's also a common concept to integrate the 2 very different kinds of systems.

1112
02:46:13.370 --> 02:46:33.519
senate chamber: The other approaches that again, we'll try and sketch. So the new symbolic approach would be to say, well, effectively, cognitive architectures are largely symbolic. Not entirely. There's some symbolic stuff, but largely, their strengths lie in the symbolic manipulation.

1113
02:46:33.520 --> 02:46:45.099
senate chamber: Large language models, generative, AI primarily subsymbolic, neural, massively parallel. There's an obvious two-layer approach that some architectures like Clarion have explored.

1114
02:46:45.100 --> 02:47:05.990
senate chamber: and finally, there is to some extent the most abstract approach. The extreme of the modular approach which is the agency approach. You can have agents that can be the cognitive architectures or generative AI. For that matter, people, and just integrate in an agent-based framework, and you don't try any kind of tight integration.

1115
02:47:07.070 --> 02:47:24.050
senate chamber: So that was a nice summary paper that laid out the general lay of the land. Now, going into more detail. We had 4 sessions that largely corresponded to 4 different ways of combining them.

1116
02:47:24.160 --> 02:47:31.800
senate chamber: and again, I won't go through the list, but you can see many of the people who are here, and who will be on the panel contributed to that

1117
02:47:33.790 --> 02:47:42.090
senate chamber: The 1st one, to some extent the most obvious one is to use Llms for knowledge generation offline.

1118
02:47:42.420 --> 02:47:58.440
senate chamber: that is, you try to extract their knowledge, and you use that knowledge as a way of providing content for cognitive architectures. In the session this morning John mentions the fact that well, cognitive models tend to be hand generated, and of course

1119
02:47:58.440 --> 02:48:13.380
senate chamber: that takes a long time, and that prevents scalability. So that's a clear path there. A couple of examples of that work by Chris Myers and Alex Hough and Othelia, and colleagues

1120
02:48:13.380 --> 02:48:34.070
senate chamber: about extracting chunks in formal form as triple subject relation objects from Llms. And use that in cognitive models that depend on knowledge. In this case an analogy type example.

1121
02:48:34.070 --> 02:48:44.109
senate chamber: And Alex has talked about that before at the workshop. That was a nice example, and the kind of difficulties that come with that.

1122
02:48:46.350 --> 02:49:15.400
senate chamber: going a little bit of a step forward is that this was very much offline extraction, suck the knowledge out, put it in the cognitive model. The model. The next step is to do it online. As the model is running a couple examples of that. So Ken Forbis has used it to augment the semantic parsers again, for the purpose of performing analogy and extract sort of richer knowledge that can be done through parsing alone.

1123
02:49:15.420 --> 02:49:40.480
senate chamber: and in particular John Laird and their colleagues have somebody mentioned earlier interactive task learning the ability to learn to perform tasks in interaction with a human or an agent, and they've explored a number of ways in which large language models a source of knowledge can be used dynamically to extract and support that kind of learning

1124
02:49:43.770 --> 02:49:54.360
senate chamber: going into sort of deeper aspects of the integration. A couple of examples. So

1125
02:49:55.070 --> 02:50:03.470
senate chamber: Rob West is on the panel, so I will mention briefly their work later.

1126
02:50:03.530 --> 02:50:28.650
senate chamber: But the 1st example, there was a piece of work that a number of us, Pete Caroni, Kevin Constantinos, Mrpoulos did, and it was on the follow up on our Covid models that was meant to simulate how populations would react would behave during an event like a pandemic.

1127
02:50:28.650 --> 02:50:41.600
senate chamber: And it seems to be a very popular pattern nowadays that's really emerging. There have been a number of papers that have just come out about using large language models for the social sciences.

1128
02:50:41.690 --> 02:50:59.900
senate chamber: So essentially, you can prompt them in such a way as to elicit the typical response of a particular subpopulation for various characteristics, and that seemed to be an emerging pattern of using Llms. To do that kind of science again, with the danger that we

1129
02:50:59.900 --> 02:51:13.200
senate chamber: we suggested in the early session. The fact that if you end up taking them as a proxy, but that the proxy doesn't really have the characteristics of humans of the real thing. You could end up being waylaid. But

1130
02:51:13.200 --> 02:51:26.270
senate chamber: still the scalability aspect and the relative ease and cheapness versus running human subject. Studies at scale, for example, is a really compelling potential use.

1131
02:51:28.160 --> 02:51:40.220
senate chamber: So again, I'll let Rob talk about his particular approach. There, on beer, on, on using generative models to essentially underlie the cognitive architecture.

1132
02:51:41.360 --> 02:52:07.569
senate chamber: Another approach was, for example, by Mike Whitbrock and colleagues, was to use large language models as effectively as the language substract for cognition. And of course the perennial controversy does language emerge from thought uses language is built on top of language. This is one particular. Take on this

1133
02:52:09.660 --> 02:52:30.720
senate chamber: The final session that called alternative architectures is a deeper unification of cognitive architectures and generative model, effectively using generative model as the overall subtract for implementation of the cognitive models.

1134
02:52:30.870 --> 02:52:57.060
senate chamber: One example was Cody Gonzalez, and colleague Cody has done a lot of instance-based learning models where the behavior is driven by experience. It's a nice brand of model, because it requires very little hand engineering. Basically, all you need to do is specify the representation of these instances, and in many experimental settings.

1135
02:52:57.060 --> 02:53:03.559
senate chamber: It's quite obvious you have very few features, and you present the features. No magic doctrine. And that's all you have

1136
02:53:03.560 --> 02:53:29.259
senate chamber: in cases where more real world cases, where it's more complex, that does raise the question of how do you select the representation. And how do you set things like similarities that drive generalization? So what Cody and colleagues did is use generative models as a way to take all that information and then output the representation for use by the instance-based learning models

1137
02:53:32.980 --> 02:53:53.800
senate chamber: Brian Magurko and colleagues looked at large language models primarily from the grounding perspective, and they had a really interesting taxonomy there, where they looked at very different kinds of grounding, and how that related to cognitive architectures and large language models.

1138
02:53:54.220 --> 02:54:17.440
senate chamber: And again, Mary Kelly had some really interesting talk about how to at a very deep level, use an abstraction of generative AI and vector, symbolic architectures as the implementation for the architecture. But again she's on the panel, and I'll let her talk about that.

1139
02:54:17.750 --> 02:54:27.240
senate chamber: There were a number of other posters, other works, and and again, some people here. Chris is on the panel, Rob, I'll let them talk about what they've done.

1140
02:54:27.400 --> 02:54:30.970
senate chamber: Oh, quick, summary here

1141
02:54:33.860 --> 02:54:54.610
senate chamber: As the meeting was going on, it started occurring to me that even though we have 2 very different form factors there between cognitive architectures and generative models, and I analogized generative models to sort of a behaviorist approach to cognition as opposed to cognitive approach.

1142
02:54:54.800 --> 02:55:03.330
senate chamber: There are a number of parallels that are quite interesting, and that sort of fundamentally could be used to underpin this kind of hybridization.

1143
02:55:03.540 --> 02:55:07.740
senate chamber: So and then you have the their number of points. So

1144
02:55:07.890 --> 02:55:28.840
senate chamber: dark green seems to be a strong agreement, and then, as you go to the orange and the red, you have more and more disagreement. But the fact, for example, that the attention mechanism is very similar to the role of working memory in cognitive architectures. The token vector distinction is very much a symbolic sub-symbolic distinction.

1145
02:55:28.920 --> 02:55:38.800
senate chamber: All these models work the output, the output are controlled by the same kind of probabilistic softmax selection that we have in the Botzmann equation.

1146
02:55:38.920 --> 02:55:55.590
senate chamber: You can go into sort of slightly more more distant analogies like the role of reinforcement. Learning the fact, interestingly enough, that those large language models sort of work in that cycle of outputting one word at a time

1147
02:55:55.590 --> 02:56:14.110
senate chamber: which how you know it's 1 10th of a second or so, which is not that different at the deliberate act level, that cognitive architectures are so sequential, processing right to have one word lead to another, and that kind of same same general sequential processing, the cognitive architectures.

1148
02:56:14.240 --> 02:56:30.720
senate chamber: But then you get into some some deep division, like the fact that generative model have no long term memory per se. It's all so baked in. And yes, you can use the context window as an extremely extended working memory. That's sort of similar to long-term memory.

1149
02:56:30.850 --> 02:56:35.680
senate chamber: but the lack of internal control, and and so on and so forth.

1150
02:56:35.870 --> 02:56:58.530
senate chamber: so wrapping up, you can sort of think at the time, and I think that that's how it partly has played out a number of ways in which the cognitive architectures and generative model could interact you effectively. And you can see it more and more. AI, exploring the space of cognitive architectures, especially as it become more agentic. AI

1151
02:56:58.780 --> 02:57:12.629
senate chamber: geez! If only people had thought about the architectures of general intelligence, and they could use that guide themselves. But who knows? It's so hard to penetrate their consciousness.

1152
02:57:12.840 --> 02:57:25.790
senate chamber: Modular integration is the low hanging fruit. There are lots of different ways of putting them together in agent tech architecture, and you can see a lot of those techniques sort of organically evolving in the AI world.

1153
02:57:26.140 --> 02:57:33.510
senate chamber: And and finally, cognitive science, cognitive psychology has evolved

1154
02:57:33.520 --> 02:58:00.960
senate chamber: a century at least of techniques for peeking into the human black box and interpreting what they're doing. Shouldn't we apply some of those techniques to do that, for example, how much, as we mentioned earlier, how much of the performance of a system like Centaur is due to the knowledge that it's ingested versus any kind of reasoning or generalization or inference that it performs.

1155
02:58:01.180 --> 02:58:30.519
senate chamber: And then, finally, Andrea, in his notes, put me next to the Barfing icon of benchmarks, and when we mentioned benchmarks in the discussion at the end of the Symposium that turned out to be a real flashpoint, where people had a spirited discussion for half an hour. That's the yin and yang of benchmarks is that we've always talked about human data and the need to model and reproduce human data.

1156
02:58:30.550 --> 02:58:49.390
senate chamber: But when you do it in sort of an atheoretical way, the way a lot of machine learning and AI does it. It's just a cynical exercise in data augmentation where you get better and better without necessarily the system itself getting better. You just pour more knowledge into it.

1157
02:58:50.110 --> 02:58:51.570
senate chamber: and I'll stop there

1158
02:58:57.490 --> 02:58:58.280
senate chamber: right.

1159
02:58:58.790 --> 02:58:59.960
senate chamber: Thanks very much.

1160
02:59:00.930 --> 02:59:05.470
senate chamber: Timer. Up in my picture.

1161
02:59:05.910 --> 02:59:12.239
senate chamber: We need to get well while the we should get the the the panel down right. Yep.

1162
02:59:12.750 --> 02:59:16.139
senate chamber: so I don't know. Chris. Nancy is going to get down. Mary.

1163
02:59:17.260 --> 02:59:25.360
senate chamber: come on down. You wanna come down, Catherine and Rob West

1164
02:59:26.570 --> 02:59:28.169
senate chamber: is Rob West gonna be here.

1165
02:59:28.390 --> 02:59:33.100
senate chamber: I think he was supposed to be online. I think I saw him online earlier.

1166
02:59:34.400 --> 02:59:35.660
Robert’s iPhone: I am online.

1167
02:59:36.490 --> 02:59:38.590
senate chamber: Oh, okay, right?

1168
02:59:39.630 --> 02:59:41.080
senate chamber: Change his name.

1169
02:59:44.490 --> 02:59:49.759
senate chamber: Okay? So the the panelists are supposed to like talk for 3 min.

1170
02:59:50.670 --> 02:59:57.650
senate chamber: We only have 4 panelists, so certainly 5 min, was the the standard. Okay, we'll give them 5 min.

1171
02:59:57.830 --> 03:00:03.680
senate chamber: And, Chris, Nancy, your turn.

1172
03:00:04.420 --> 03:00:07.920
Christopher Dancy: Alright, I'll go first.st let me.

1173
03:00:08.420 --> 03:00:10.913
Christopher Dancy: I was gonna put a stop with just case

1174
03:00:11.370 --> 03:00:16.459
Christopher Dancy: So glad to be here. Thank you, Christian and Frank for

1175
03:00:16.820 --> 03:00:21.750
Christopher Dancy: keeping us in line through all this. So

1176
03:00:22.290 --> 03:00:37.419
Christopher Dancy: for thinking through this, I don't have a slide. I didn't really want to do slides. Just kind of a set of notes. To keep the talking points. When I started thinking about this idea and thinking through more. So

1177
03:00:38.220 --> 03:00:53.230
Christopher Dancy: mixed representations. Really came from for me the standard model of the mind symposium before. Where we had this, you know, nice session on knowledge level representations and some real questions coming up on

1178
03:00:53.560 --> 03:00:56.489
Christopher Dancy: better representations of those things

1179
03:00:56.920 --> 03:01:24.489
Christopher Dancy: within these architectures, and that being a missing thing, and I started to take that seriously, because at the same time, in parallel, I was really reading a lot of social critical theory and thinking about the social world and the way it impacts behavior and thinking. And how can we represent that in in cognitive systems and use incognitive architectures and really use the power we have mechanistic models.

1180
03:01:25.590 --> 03:01:26.723
Christopher Dancy: the guard.

1181
03:01:28.030 --> 03:01:35.668
Christopher Dancy: to to really explain certain phenomena in ways that make sense for us and for me. So it's pretty

1182
03:01:36.670 --> 03:01:41.890
Christopher Dancy: pretty selfish in that ways. And so

1183
03:01:42.230 --> 03:02:07.350
Christopher Dancy: kind of that that went forward. I was thinking about originally in terms of systems like conceptnet bigger systems that we could use for kind of these knowledge sources. And eventually Llms grew, and and I thought that might be an interesting place in terms of generative systems, generative models, and think about the connections there. And so so in the original paper, I kind of just talked about a few things, a lot of what Christian talked about.

1184
03:02:07.727 --> 03:02:20.559
Christopher Dancy: Think, of course, of that low hanging fruit, right? Generative models as social, cultural symbol generators, maybe for cognitive architectures. And I'm particularly worried about social level perspectives. Right. How do we represent

1185
03:02:20.730 --> 03:02:45.680
Christopher Dancy: latent social models of the world within architectures? Given that, we know they're there right? You might think about this, I think sometimes as implicit bias. I'm not sure that's always the most useful framing. In fact, I'm sure it's not often the most useful framing, but as a way to think about it right? How do we represent those structures in memory at scale, in a way that makes sense because it matters for our models? Right? If we're going to really think about them realistically in these in these worlds

1186
03:02:45.900 --> 03:02:48.810
Christopher Dancy: and impactful ways that that

1187
03:02:49.030 --> 03:03:09.849
Christopher Dancy: I wouldn't say scale. But generalize a little bit better because we operate within these social worlds, and the other part, I think potentially the other, maybe lower hanging fruit. If you make some of these connections, particularly with Llms. Given. How much they're used now, is the potential for using cognitive models and cognitive architectures is kind of

1188
03:03:10.270 --> 03:03:24.880
Christopher Dancy: auditing tools for some of these models. If you wanted to think about it that way as well. To really, for me, it's it's thinking, through what does it mean? Given the world we're in now to have people use Llms as main knowledge sources.

1189
03:03:25.110 --> 03:03:48.149
Christopher Dancy: Right? What does that mean cognitively? What does that mean for people? And why might that be a bad idea. I think there are many ways that we might think and know that it's a bad idea implicitly, but I think getting those mechanistic models for understanding where there's a bad idea, maybe where it might be useful, would be useful for us beyond just kind of the hype of the models themselves, which there is. I'm sure everybody knows a lot of hype that's not true.

1190
03:03:48.793 --> 03:04:13.456
Christopher Dancy: And then the other big thing that I thought was worth mentioning. Is, it's been kind of mentioned before with these generative models. Potentially, these types of systems as we interact with them, the problem with scale, the problem with size and how much, how many environmental resources that really takes. I think that's a consideration we always have to have. As we're thinking about this, and also within those knowledge sources, who and what is represented sources and how they might be problematic.

1191
03:04:13.780 --> 03:04:32.469
Christopher Dancy: particularly given that we don't even know they're not sources of some of the generative models that might be useful or might be used. So I tend to advocate for completely open models ones that you know the data as well as the architecture, which is why I don't like llama in some ways, because they're not open about their data for

1192
03:04:32.860 --> 03:04:35.099
Christopher Dancy: probably certain reasons.

1193
03:04:35.824 --> 03:04:49.540
Christopher Dancy: But the other thing to say is, if you understand that the Llms are problematic, that also allows you to explore certain social phenomena in an interesting way as well. So if you know it has problems in terms of racialization.

1194
03:04:49.670 --> 03:04:56.890
Christopher Dancy: what does that mean to have cognitive models have problems? There's racialization. And how does that affect the way to interact with the world? So that's

1195
03:04:57.080 --> 03:05:02.320
Christopher Dancy: that's my plug for thinking about this and think about these things at scale. I'll stop there.

1196
03:05:04.800 --> 03:05:06.730
senate chamber: Okay, thank you very much. Both.

1197
03:05:07.940 --> 03:05:09.760
senate chamber: Thank you very much.

1198
03:05:09.880 --> 03:05:14.580
senate chamber: Any one comment or query to that feature

1199
03:05:18.530 --> 03:05:29.420
senate chamber: bye, yeah, want to make a brief comment on this idea. Under that this might be.

1200
03:05:29.520 --> 03:05:31.009
senate chamber: don't have any fruits.

1201
03:05:32.420 --> 03:05:40.940
senate chamber: Given. The potential for amplifying these laden social models were biases.

1202
03:05:41.810 --> 03:05:45.499
senate chamber: It might actually be low hanging in the brochure.

1203
03:05:47.600 --> 03:05:59.869
Christopher Dancy: No, I mean, I agree right? And that's why I think if you're clear about it, you can use it to understand that there are parts of the world that is like that. And what does that mean? Right for for certain things? So I think

1204
03:06:00.090 --> 03:06:21.409
Christopher Dancy: if you think about it as a perfect model of the world. You run into problems. If you recognize it for the problematic model it is, you can maybe explore some problematic social theory as well, which it's critically at least. But yeah, I agree that you have to be very careful. And and that idea of engineering pipelines can be particularly problematic, because it can allow people to do things that you maybe don't want them to do.

1205
03:06:25.852 --> 03:06:30.660
senate chamber: Okay? So I believe you're nuts.

1206
03:06:37.530 --> 03:06:38.390
senate chamber: something across this.

1207
03:06:40.230 --> 03:06:43.219
senate chamber: Okay? So we actually have one slide to share.

1208
03:06:44.020 --> 03:06:45.280
senate chamber: Oh.

1209
03:06:49.690 --> 03:06:50.660
senate chamber: my dear, okay,

1210
03:06:52.740 --> 03:06:58.690
senate chamber: So Christian kindly mentioned my particular approach, which is vector symbolic approaches.

1211
03:06:58.830 --> 03:07:00.510
senate chamber: This is just one in the

1212
03:07:00.670 --> 03:07:09.219
senate chamber: very large umbrella under symbolic approaches. And in a vector symbolic approach. We are building the symbols directly into vector representations.

1213
03:07:10.510 --> 03:07:11.820
senate chamber: operate over.

1214
03:07:12.610 --> 03:07:14.170
senate chamber: And

1215
03:07:14.290 --> 03:07:23.969
senate chamber: but at least, broadly speaking, I do think neurosymbolic approaches may be the solution to a lot of our problems. I'm going to focus on a very quickly talk about a couple of problems that

1216
03:07:24.440 --> 03:07:27.889
senate chamber: my lab has been very interested in, kind of trying to work on

1217
03:07:28.180 --> 03:07:33.630
senate chamber: one of these problems has been talked about already today, which is on memory and learning.

1218
03:07:34.260 --> 03:07:38.720
senate chamber: So the problem closer to the microphone. Oh, sorry like this. Close this close.

1219
03:07:39.910 --> 03:07:41.850
senate chamber: So one of the problems that we're in

1220
03:07:42.610 --> 03:07:48.839
senate chamber: visiting today has been efficient online continuous learning that scales

1221
03:07:49.120 --> 03:07:56.020
senate chamber: actar has online and continuous. It's got this great episodic memory retrieval going on.

1222
03:07:56.170 --> 03:07:58.809
senate chamber: But we, you know, it's it's got some scaling issues.

1223
03:07:58.920 --> 03:08:09.720
senate chamber: Meanwhile the transformer based models like the large language models are quite efficient. Well, maybe not as efficient as the human brain. There's a talk

1224
03:08:10.230 --> 03:08:14.000
senate chamber: during this conference how

1225
03:08:14.120 --> 03:08:32.170
senate chamber: babies learn language on much less data, much much much less data than these large language models. But they definitely scale. We can definitely say that they scale really well. But they are not online learners. They don't continuously learn.

1226
03:08:32.430 --> 03:08:34.559
senate chamber: they don't do episodic memory.

1227
03:08:34.670 --> 03:08:44.950
senate chamber: This is not a problem I have solved. But it's a problem that I've tried to solve. So I posed a couple of different models, holographic declarative memory and cog engine

1228
03:08:46.010 --> 03:08:55.600
senate chamber: and Mira Range actually had a talk today about not today. Yesterday, extending my holographic record of memory work, which was cool.

1229
03:08:57.310 --> 03:09:02.139
senate chamber: But yeah, it turns out, it's still a hard problem, getting something that is efficient. Scales

1230
03:09:02.260 --> 03:09:10.580
senate chamber: learns online and continuously does all of these things at once. But I do think neurosymbolic approaches might be the way forward to get the best of both worlds here.

1231
03:09:12.140 --> 03:09:18.590
senate chamber: The another problem that I think we've also touched on today is reliable problem solving

1232
03:09:19.200 --> 03:09:27.560
senate chamber: actar. You can definitely set up Aktar to be good at doing some sort of formal problem, solving on some problem and doing that very reliably. But it won't be. It'll be like

1233
03:09:27.660 --> 03:09:31.559
senate chamber: handcrafted for a certain problem. Space will be very general.

1234
03:09:31.700 --> 03:09:34.299
senate chamber: These transformer based models are quite

1235
03:09:34.430 --> 03:09:49.110
senate chamber: general, but they are not reliable. And when I say they're general, they are general to things in their training data. What's your outside of their training data? Some some sort of very new tests. They're not going to do. Well.

1236
03:09:49.560 --> 03:09:56.239
senate chamber: So my Phd students, Eileen has been working on trying to create

1237
03:09:56.360 --> 03:10:10.589
senate chamber: a language sort of a, if you will, a language of thought for vector symbolic reasoning that would allow for very general problem solving in a formal way that potentially neural networks could learn to do.

1238
03:10:12.250 --> 03:10:23.309
senate chamber: Certainly this is a problem that Deepind and Openai are trying to tackle. But we're not exactly sure what they're doing, because they're not open with their architectures or their data sets.

1239
03:10:25.330 --> 03:10:29.340
senate chamber: Another thing is just reasoning about the world and other minds.

1240
03:10:30.710 --> 03:10:36.100
senate chamber: One of my students has. In fact, this is very relevant to Centaur. We didn't have Centaur, but

1241
03:10:36.250 --> 03:10:42.989
senate chamber: one of my students tried testing out large language models on their ability to do theory of mind.

1242
03:10:43.310 --> 03:10:54.999
senate chamber: And the short answer is, they can't do it. They can do it. If you give them a theory of mind task. That's from the theory of mind literature. In which case, yeah, they can do it. If you modify that task slightly, it fails.

1243
03:10:55.190 --> 03:11:17.880
senate chamber: And if you give it a real world theory of mind task like we give it a cooperative card game where there's 2 players and one of the players is a large language model, or both players. A large language model doesn't really matter, and they have to reason about the other player, and they can't. They can't do it. They can't reason about the other player to achieve a joint cooperative goal.

1244
03:11:18.350 --> 03:11:25.709
senate chamber: And what's what is a system that might allow us to model, how things behave according to goals and actions, and so on.

1245
03:11:25.820 --> 03:11:29.379
senate chamber: Well, it certainly is the large language model.

1246
03:11:31.060 --> 03:11:48.400
senate chamber: so we don't have a proposed architecture there, but I do think theory of mind is a place where there could be a very fruitful fusion of the insights of generative models. The generality of general models, if you will, and then the reasoning capacities of conventional cognitive architectures.

1247
03:11:49.970 --> 03:11:54.080
senate chamber: Okay, we have one comment while we set up

1248
03:11:56.880 --> 03:12:01.980
senate chamber: might already be set up. But there's 1, no new guy.

1249
03:12:02.820 --> 03:12:12.290
senate chamber: Yes. So the comment is this technology, the Llms in France they develop literally by the month.

1250
03:12:13.010 --> 03:12:20.449
senate chamber: and it is dangerous to criticize that they cannot do something. There is a long record now of people saying they cannot do X,

1251
03:12:20.570 --> 03:12:23.350
senate chamber: and next month doing good.

1252
03:12:23.510 --> 03:12:37.640
senate chamber: And so when we talk, the comment is, try to kind of extrapolate and think about a class of models, and not the specific instance that you tested. I know it is a uncomfortable place to be, but

1253
03:12:37.850 --> 03:12:40.039
senate chamber: with something that's moving so fast.

1254
03:12:43.330 --> 03:13:03.670
senate chamber: I do think like what we were talking about with Centaur. It's largely, just kind of statistically fitting to the data set. So their current solution is bigger models, larger data sets. And you're getting a better statistical fit to the data set. I don't think that's necessarily ever going to get you to things like theory of mind, because it just doesn't have the right mechanisms

1255
03:13:08.360 --> 03:13:11.490
senate chamber: next panelist. Please. All right.

1256
03:13:12.020 --> 03:13:16.304
senate chamber: yeah. Sorry about the microphone. I also don't have any slides, and I'm gonna wing it a little bit.

1257
03:13:16.910 --> 03:13:22.360
senate chamber: particularly because the bulk of my research comes with this problem ways

1258
03:13:22.700 --> 03:13:25.840
senate chamber: and that it doesn't touch the lens at all.

1259
03:13:27.020 --> 03:13:31.890
senate chamber: The main goal is to validate

1260
03:13:32.170 --> 03:13:39.180
senate chamber: structures proposed by architectures in the brain, because it would be very nice to be able to say that our

1261
03:13:39.390 --> 03:13:44.610
senate chamber: symbolic conceptions of how cognition actually works.

1262
03:13:44.820 --> 03:13:51.970
senate chamber: Attendees echoed in what we understand about brain signals and biology.

1263
03:13:52.910 --> 03:13:57.259
senate chamber: and to that effect. Most of what I presented at this

1264
03:13:57.890 --> 03:14:03.119
senate chamber: workshop was about how we had been able to at least initially draw

1265
03:14:03.240 --> 03:14:11.380
senate chamber: a connection between the structure as proposed by a common model of cognition and brain signaling analysis.

1266
03:14:12.430 --> 03:14:34.520
senate chamber: But it sort of occurs to me that a lot of these solutions that we talked about in the workshop were, to some degree or another, proposing to use architectures and sort of the structures that they represent as a means of containing or constraining or otherwise shaping the types of things that Llms or other generate models might produce.

1267
03:14:35.210 --> 03:14:52.460
senate chamber: And if you're willing to squint a little bit, this is basically what we were doing to make those connections between architectures and brain data. We were using an architectural structure as a means of constraining a generative model, not a language model.

1268
03:14:52.610 --> 03:14:56.149
senate chamber: a generative model that was trying to predict brain signals.

1269
03:14:58.010 --> 03:15:01.630
senate chamber: And from that perspective our main finding is that

1270
03:15:01.850 --> 03:15:08.609
senate chamber: implementing such a connection is pretty straightforward. But the key word is an implementation.

1271
03:15:09.120 --> 03:15:21.030
senate chamber: and what turns out to be much more challenging is saying anything concrete about whether or not that implementation is better, worse, more representative of anything than any other.

1272
03:15:21.210 --> 03:15:35.049
senate chamber: So we're able to do this to a certain extent. But when we tried to get deeper into the details, it became really challenging to figure out how any particular change in the structure that we were making was impacting

1273
03:15:35.970 --> 03:15:49.010
senate chamber: that was being produced. And we've already touched on it today that this, I think, is a fundamental challenge with models like this is that if your goal is to say we're going to put these 2 methods together, and it's going to make them better.

1274
03:15:49.660 --> 03:15:53.240
senate chamber: How do you measure that? What does it mean to be better?

1275
03:15:53.510 --> 03:15:56.950
senate chamber: So I guess my question to

1276
03:15:57.510 --> 03:16:01.630
senate chamber: the room is. But is this something that we anticipate

1277
03:16:01.780 --> 03:16:08.269
senate chamber: when we? So that these problems that I have are specific to my type of generative model? It's not a 1 to one. Comparison.

1278
03:16:08.790 --> 03:16:14.570
senate chamber: Means of generation is different. What is predicting is different, but I think there is still

1279
03:16:15.620 --> 03:16:22.529
senate chamber: high potential that we will run into the same class of issues with interpreting and evaluating what is predicted when we

1280
03:16:23.160 --> 03:16:26.940
senate chamber: try, and architectures and language models

1281
03:16:28.300 --> 03:16:36.980
senate chamber: needs to be addressed, and there are, do we? Do we have a plan or an approach, or how to think about those issues?

1282
03:16:40.540 --> 03:16:41.580
senate chamber: Thank you very much.

1283
03:16:42.720 --> 03:16:44.820
senate chamber: One comment, or

1284
03:16:49.610 --> 03:16:52.400
senate chamber: sure I'll take a quick stab.

1285
03:16:53.260 --> 03:16:58.859
senate chamber: So so, looking at cognitive architectures themselves

1286
03:17:00.740 --> 03:17:07.629
senate chamber: purely symbolic models, it's always easy to tell what they're doing, because that's an advantage of symbol.

1287
03:17:07.770 --> 03:17:18.640
senate chamber: But when you look at a broad architecture like Aktar, 2 things that have served us well. Beyond that there is one is the modular structure of the architecture.

1288
03:17:18.890 --> 03:17:36.529
senate chamber: The fact that even if the individual pieces were black boxes, you can tell that's the old joke about the brain. Once neuroimaging came along, went from being one black box to 49 black boxes, or however many, right? But at least it tells you something about the structure of code, and you need a lot of the

1289
03:17:36.650 --> 03:17:58.250
senate chamber: the analyses that can be done. Like, for example, Andrea's validation of the common model against human connector data is all about that structure of processing. And you think that's a technique that could generalize to this kind of hybrid systems hybridization, or at least taking the modular approach.

1290
03:17:58.280 --> 03:18:07.299
senate chamber: And the second thing that has served as well, and we've used it in a number of recent projects, like, for example, the explainable layout project.

1291
03:18:07.300 --> 03:18:31.359
senate chamber: where, even though the subsymbolic level of the architecture you can introspect. It's tractable enough to be able to introspect, figure out, even though the behavior, the output of some module, is the result of complex computation. You can analyze those computations in such a way that you can introspect in a way that is very, very difficult to do, not impossible, but very difficult

1292
03:18:31.360 --> 03:18:36.330
senate chamber: able to do in those very deep, transformer architectures.

1293
03:18:36.360 --> 03:18:44.120
senate chamber: So and again, I think that's the kind of thing that could be leveraged to to try and understand this hybrid.

1294
03:18:45.530 --> 03:18:52.369
senate chamber: Yeah? And I definitely think that incorporating architecture type models will help

1295
03:18:52.770 --> 03:18:59.269
senate chamber: with some of the problems that you get from trying to interpret a large language model alone, but to the modular approach.

1296
03:18:59.970 --> 03:19:06.620
senate chamber: What I've been working on for the last couple of years has been trying to extend that initial result

1297
03:19:06.940 --> 03:19:12.289
senate chamber: look deeper into it. And that's where we're running into problems that we can very broadly say something.

1298
03:19:13.140 --> 03:19:23.000
senate chamber: And I do think that that approach can be scaled and and applied in other places, but a another level of detail below that

1299
03:19:23.260 --> 03:19:25.780
senate chamber: proved much harder to PIN down.

1300
03:19:29.780 --> 03:19:32.009
senate chamber: Okay, thank you.

1301
03:19:32.450 --> 03:19:37.290
senate chamber: We're about to go to Rob West. Are you ready along the canal there, Rob.

1302
03:19:37.760 --> 03:19:40.480
Robert’s iPhone: Yes, actually, I'm in Vancouver.

1303
03:19:41.760 --> 03:19:44.870
Robert’s iPhone: But yeah, can you hear me?

1304
03:19:47.040 --> 03:19:47.590
senate chamber: Yes.

1305
03:19:47.990 --> 03:19:48.969
Robert’s iPhone: You can hear me.

1306
03:19:49.110 --> 03:19:49.630
senate chamber: You're.

1307
03:19:49.630 --> 03:19:53.683
Robert’s iPhone: Okay, great, great, great. Okay? So very quickly.

1308
03:19:54.980 --> 03:19:58.049
Robert’s iPhone: yeah, I think we definitely need something.

1309
03:19:58.190 --> 03:20:03.860
Robert’s iPhone: a way to interface between these 2 different systems. I think Mary described it really well how they're different.

1310
03:20:04.030 --> 03:20:06.869
Robert’s iPhone: We need some way to go between them

1311
03:20:07.130 --> 03:20:14.579
Robert’s iPhone: if we want to do hybrid approaches, and I personally think the holographic approach is very good. It could be other ways or

1312
03:20:14.720 --> 03:20:16.829
Robert’s iPhone: different holographic approaches.

1313
03:20:17.070 --> 03:20:24.310
Robert’s iPhone: But so now the the proposal that I made

1314
03:20:25.169 --> 03:20:32.410
Robert’s iPhone: it's a little bit different. And I was thinking about how to say it. Simply. So I'm gonna put it in terms of declarative memory.

1315
03:20:32.650 --> 03:20:36.890
Robert’s iPhone: So I think possibly one of the things we need to do

1316
03:20:37.160 --> 03:20:40.600
Robert’s iPhone: is question. Some of the deep, deep

1317
03:20:41.240 --> 03:20:44.530
Robert’s iPhone: assumptions of act are that never really get tested.

1318
03:20:44.810 --> 03:20:53.249
Robert’s iPhone: So we have working memory and declarative memory that comes just straight from psychology. And certainly there's phenomena consistent with those.

1319
03:20:53.600 --> 03:20:59.170
Robert’s iPhone: But basically what I proposed was something called middle memory, basically saying that

1320
03:20:59.430 --> 03:21:11.550
Robert’s iPhone: what we take to be declarative memory or long term, you know, drawing right from long term, memory in act R is not. It's something created on the fly. And that's 1 way

1321
03:21:11.800 --> 03:21:22.410
Robert’s iPhone: to incorporate large language models is that they're constantly dumping in. So you need a mechanism to create it on the fly and large language models can provide part of that.

1322
03:21:22.560 --> 03:21:24.629
Robert’s iPhone: So, but that's a that's a deep.

1323
03:21:24.960 --> 03:21:30.229
Robert’s iPhone: So that's that's point number one is, I think we need to consider

1324
03:21:31.280 --> 03:21:44.110
Robert’s iPhone: questioning some of these things. Not in a way that questions. The results of act are. But maybe, like memory is a big thing. Maybe there's a, you know, like some people think declarative memory is

1325
03:21:44.460 --> 03:21:50.299
Robert’s iPhone: semantic and episodic. Some people think we need a different episodic. So it's a discussion along those lines. Do we need

1326
03:21:50.400 --> 03:21:56.500
Robert’s iPhone: something like, I'm I'm questioning the hierarchy. Do we need something in between working memory

1327
03:21:56.700 --> 03:22:07.509
Robert’s iPhone: and long term storage memory? And that is a place where I propose that large language models could play a fairly big role in populating that in a dynamic, ongoing way.

1328
03:22:08.000 --> 03:22:10.739
Robert’s iPhone: Okay, so that's that idea. It's in the paper.

1329
03:22:11.180 --> 03:22:17.130
Robert’s iPhone: The other thing I wanted to bring up, though, was questioning.

1330
03:22:17.720 --> 03:22:20.259
Robert’s iPhone: And I think this has come up in a lot of talks.

1331
03:22:20.650 --> 03:22:28.370
Robert’s iPhone: Do large language models actually model a part of human cognition like legitimately so

1332
03:22:29.170 --> 03:22:31.990
Robert’s iPhone: right? So like Mary gave the example of

1333
03:22:32.440 --> 03:22:36.410
Robert’s iPhone: theory of mind. So they're bad at theory of mind. They're not.

1334
03:22:36.850 --> 03:22:42.629
Robert’s iPhone: Not that they can't do it. They're bad at, but some people are bad at it, and some people, when they're in a hurry don't do it right?

1335
03:22:42.930 --> 03:22:45.779
Robert’s iPhone: So at some level.

1336
03:22:46.660 --> 03:23:08.149
Robert’s iPhone: I think we need. And this is this is a controversial thing, right? Because even in the AI community there's a lot of controversy about this, like the Koch paper and the recent apple paper. Just sort of saying it's just mimicking. It's just pulling from examples. And then other people say, you know, some interesting work, actually looking at imaging the neural networks

1337
03:23:08.450 --> 03:23:23.560
Robert’s iPhone: and finding there's evidence of cognition, actually storing information and using it later, very deliberately. That's interesting. Now the thing is, is it like a human doing that? Does it learn language like a human? No.

1338
03:23:23.710 --> 03:23:32.499
Robert’s iPhone: But once it gets to a certain point, does it? The question is, does it arrive somewhere by a completely different means, to a place that is actually similar to us?

1339
03:23:32.940 --> 03:23:35.409
Robert’s iPhone: And then we can turn that around. And we can ask

1340
03:23:35.880 --> 03:23:39.810
Robert’s iPhone: you humans not all the time, not when they're thinking carefully.

1341
03:23:39.930 --> 03:23:42.730
Robert’s iPhone: but when they're just talking like we're all talking now.

1342
03:23:43.290 --> 03:23:47.700
Robert’s iPhone: or when we're just sort of spitballing it, you know, coming up with ideas.

1343
03:23:47.860 --> 03:23:49.790
Robert’s iPhone: tossing things to back and forth.

1344
03:23:50.710 --> 03:23:57.320
Robert’s iPhone: Are we actually doing what large language models are doing like, and I don't know.

1345
03:23:57.720 --> 03:24:05.860
Robert’s iPhone: but I think it's an important question to ask, because I hear people going back and forth on whether they have anything to do with cognition or not.

1346
03:24:06.788 --> 03:24:09.380
Robert’s iPhone: And my suggestion, I think

1347
03:24:10.160 --> 03:24:13.279
Robert’s iPhone: that humans do act like large language models

1348
03:24:13.430 --> 03:24:17.789
Robert’s iPhone: when they're, you know, just in a certain flow mode, you know, talking and so on.

1349
03:24:17.900 --> 03:24:19.940
Robert’s iPhone: But that's that's what I think

1350
03:24:20.210 --> 03:24:22.393
Robert’s iPhone: could argue about that. But

1351
03:24:24.070 --> 03:24:39.460
Robert’s iPhone: I think a way to. To maybe conceptualize it in terms of a hybrid architecture, would to say, would be to say that large language models don't really tell us much about the cognitive level. But they do operate a lot like us at the knowledge level.

1352
03:24:39.940 --> 03:24:44.060
Robert’s iPhone: So one way to think about it is

1353
03:24:44.570 --> 03:24:46.639
Robert’s iPhone: from a hybrid point of view is

1354
03:24:46.900 --> 03:24:54.440
Robert’s iPhone: we do something like we chat away and we do something. Actor is not good at modeling that large language models are

1355
03:24:55.397 --> 03:25:00.209
Robert’s iPhone: can we just sort of plug them in sort of at the knowledge level.

1356
03:25:00.750 --> 03:25:04.209
Robert’s iPhone: And sort of say, this gets done. We're not sure

1357
03:25:04.350 --> 03:25:07.629
Robert’s iPhone: how it gets done, even in large language models. But

1358
03:25:08.170 --> 03:25:12.030
Robert’s iPhone: anyway, that's the that's what I wanted to bring up this issue of.

1359
03:25:12.220 --> 03:25:17.059
Robert’s iPhone: because I hear different opinions in the audience about whether large language models?

1360
03:25:17.490 --> 03:25:20.350
Robert’s iPhone: One can they think 2.

1361
03:25:20.700 --> 03:25:27.370
Robert’s iPhone: Do they tell us anything? Is there any parallel with what humans do? So I think those are 2 questions that are really important for us.

1362
03:25:28.070 --> 03:25:30.670
Robert’s iPhone: and I'll just I'll stop there.

1363
03:25:31.690 --> 03:25:37.670
senate chamber: Thank you very much. Take one directed towards rob question comment and then open it up.

1364
03:25:42.140 --> 03:25:50.020
senate chamber: Oh, yeah, go on, hey, Rob, I'm here for American nation.

1365
03:25:51.070 --> 03:25:51.960
senate chamber: What

1366
03:25:52.600 --> 03:26:01.660
senate chamber: what do you mean by they behave like the knowledge level, like we've got all renewal, I assume, but like, can you? Can you be a little bit more precise about like what.

1367
03:26:01.660 --> 03:26:02.340
Robert’s iPhone: Yeah.

1368
03:26:02.730 --> 03:26:05.110
senate chamber: Say that that, like the emails.

1369
03:26:05.580 --> 03:26:28.150
Robert’s iPhone: Okay, yeah, I'll be much more precise. So Newell said, the knowledge level is basically about predicting other agents. Right? So it's like, same as Dennett's intentional level. So we attribute things to other agents. And we think this guy's gonna do this next, this next, this next. So instead of next word is next action from the new point of view. But if we just allow it to also be next word.

1370
03:26:28.830 --> 03:26:32.709
Robert’s iPhone: then that would just be.

1371
03:26:33.420 --> 03:26:37.740
Robert’s iPhone: And we know people do this when you're listening to me, you're predicting my next word

1372
03:26:38.250 --> 03:26:42.979
Robert’s iPhone: right? Otherwise you're unable to hear it for the most part.

1373
03:26:43.310 --> 03:26:46.910
Robert’s iPhone: but beyond that if we are doing that at times.

1374
03:26:47.220 --> 03:27:02.070
Robert’s iPhone: the question is, how much thinking gets done in this flowy sort of word, talking way like there's no doubt that large language models can do certain tasks that we thought could not be done by talking, such as playing chess.

1375
03:27:02.410 --> 03:27:03.770
Robert’s iPhone: and maybe

1376
03:27:03.910 --> 03:27:09.860
Robert’s iPhone: and I've talked to some people who are very good chess players who I've been playing against, and they're like it is not just

1377
03:27:10.630 --> 03:27:13.400
Robert’s iPhone: they. They believe it's coming up with original stuff.

1378
03:27:14.930 --> 03:27:16.629
Robert’s iPhone: You know, it's good at programming.

1379
03:27:17.150 --> 03:27:19.600
Robert’s iPhone: It's it's good at a lot of things. So our.

1380
03:27:19.700 --> 03:27:21.300
Robert’s iPhone: I guess what I'm saying is.

1381
03:27:22.280 --> 03:27:35.310
Robert’s iPhone: yeah, that that is what I'm saying that. So at the knowledge level, in that, it's predicting what comes next, predicting what comes next, and that that type of system is surprisingly more intelligent than we thought it was. It's it's capable

1382
03:27:35.450 --> 03:27:38.489
Robert’s iPhone: of a lot more intelligent behavior

1383
03:27:38.820 --> 03:27:43.309
Robert’s iPhone: than we originally thought. We we used to think. Oh, we have to think it, then say it

1384
03:27:43.470 --> 03:27:46.879
Robert’s iPhone: for the most part, but so it would be challenging. That

1385
03:27:50.160 --> 03:27:52.570
Robert’s iPhone: is that answer it no.

1386
03:27:54.090 --> 03:27:58.039
senate chamber: Right. Okay. You're you got slightly muted at the end. But.

1387
03:27:58.040 --> 03:27:59.730
Robert’s iPhone: Oh, okay, sorry. Yeah.

1388
03:28:01.620 --> 03:28:06.449
Robert’s iPhone: Probably the end was just me repeating myself like a like a language model.

1389
03:28:12.290 --> 03:28:19.109
senate chamber: Yeah, I want to repeat what Rob said one more time, but just to point out that you'll remark

1390
03:28:19.270 --> 03:28:23.130
senate chamber: that the details of the architecture get lost.

1391
03:28:23.750 --> 03:28:24.520
Robert’s iPhone: Thank you.

1392
03:28:25.020 --> 03:28:25.700
Robert’s iPhone: So.

1393
03:28:26.960 --> 03:28:28.020
senate chamber: Some people.

1394
03:28:28.980 --> 03:28:29.579
Robert’s iPhone: I know.

1395
03:28:30.700 --> 03:28:35.229
senate chamber: So you want to tell us that we have about 25 min of discussion.

1396
03:28:36.920 --> 03:28:48.109
senate chamber: That's not what I was about to tell you. But yes, I can tell you that. I just want to keep the organizer. Yeah. So so this is open discussion. But one thing I want to mention about the

1397
03:28:48.620 --> 03:29:18.279
senate chamber: predicting the other agents, and that goes back in that social simulation direction. There was a recent paper where they gave large language models, a two-hour interview with a number of human subjects, and think of it as that Npr. Interview where you have people sort of recounting their life, and from that two-hour interview.

1398
03:29:18.280 --> 03:29:39.070
senate chamber: they were able to predict the subject's response to a number of like survey type questions, things big, 5 game theory problems, etc. Etc. With 85% accuracy and basically as accurate as a given subject is going to be self-reliable.

1399
03:29:39.150 --> 03:30:07.040
senate chamber: So on the one hand, that's quite a remarkable result. I mean, basically, evidence can listen to you, Frank, for 2 h and predict that you're going to mention chickens and this and that. On the other hand, you know, you can look at that and say, well, that is just statistical patterns, right? I can tell from the history of your life where you're going to fit within the Big 5, or what your approach to game theory is going to be, and all that.

1400
03:30:07.040 --> 03:30:35.809
senate chamber: and that, as John said, that doesn't reflect anything on the architecture that is just at the behavioral level, picking up patterns of behavior. On the other hand, that's very much something that can be powerful. It depends what your. If your goal is understanding the nature of human cognition, that's not really all that useful. If your goal is having sort of a cheap ability to simulate the societies at scale, or

1401
03:30:35.810 --> 03:30:53.190
senate chamber: some widget that for whatever reason can exploit that prediction to do something useful, then it's obviously a serious advance. So I think it's going to behoove to us to figure out, where do we fit in that world? Right? And how can we exploit what

1402
03:30:53.190 --> 03:31:02.320
senate chamber: clearly a new type of artifact that could be useful. One thing, so, speaking of usefulness, one thing I want to mention is that

1403
03:31:02.320 --> 03:31:23.739
senate chamber: so John and Dan have been using large language models to try and develop, have them model Akta, can you give them a problem and create an actar model? That would be an obvious way in which those capabilities could serve our purposes. I don't know if John and Dan want to comment of where they are in that respect, and how they do better.

1404
03:31:24.410 --> 03:31:28.519
senate chamber: I'm not serious work, but Dan can give a more accurate report in any case.

1405
03:31:30.680 --> 03:31:33.719
senate chamber: Damn damn, yeah. So I guess

1406
03:31:34.280 --> 03:31:40.960
senate chamber: from my perspective, it hasn't done very well. They can produce models for some of the tutorial tasks.

1407
03:31:41.080 --> 03:31:47.350
senate chamber: probably because the actar tutorial is available, and they've been able to get access to those models

1408
03:31:47.470 --> 03:31:50.290
senate chamber: we did find actually that

1409
03:31:51.050 --> 03:32:00.599
senate chamber: sometimes they could get close. They could like you could even novel task when it wasn't in the tutorial. In fact, John asked Chat Gpt to create a task

1410
03:32:00.820 --> 03:32:03.180
senate chamber: which we then fed to some of the other models.

1411
03:32:03.450 --> 03:32:10.420
senate chamber: the other Llms. To see if they could produce an actor model and code to perform the task. And

1412
03:32:11.300 --> 03:32:19.100
senate chamber: I guess Claude actually almost did it. It produced code that almost ran, and it produced a model that could almost do the task.

1413
03:32:19.665 --> 03:32:33.160
senate chamber: It took me very little time to fix those problems. I couldn't get it to fix its own problems, but for me it took me about 10 min to fix that code and model and make them work.

1414
03:32:33.400 --> 03:32:41.550
senate chamber: It didn't do the task well, but it succeeded. The code ran, and the model ran so

1415
03:32:42.770 --> 03:32:50.679
senate chamber: I don't think they're great at Actar code, but they could maybe give you a bootstrap at some level if you know what you're doing. That's the thing, though.

1416
03:32:51.190 --> 03:32:57.380
senate chamber: If you don't know how to write an accurate model, you're probably not going to be able to do anything with what it gives you.

1417
03:32:57.530 --> 03:32:58.850
senate chamber: If it doesn't work.

1418
03:33:01.680 --> 03:33:04.659
senate chamber: I think part of the problem here is.

1419
03:33:05.210 --> 03:33:15.570
senate chamber: I'm sure, just a fraction of the problem is, there isn't the same population of actar models as there are of python code, etc.

1420
03:33:15.810 --> 03:33:19.780
senate chamber: It's a very thin database to run from.

1421
03:33:25.260 --> 03:33:29.479
senate chamber: No, they're talking to you. Go ahead. Okay, this is for Rob. So

1422
03:33:29.690 --> 03:33:33.670
senate chamber: so, Rob. But I think you're saying I'm going to reinterpret. It is that he that

1423
03:33:37.200 --> 03:33:38.520
senate chamber: that if if

1424
03:33:39.070 --> 03:33:54.150
senate chamber: if large language models or that kind of architecture. This transformer is really capturing human behavior to some degree. I think your comment was trying to summarize is that there might be immersion properties to be able to understand it's doing interesting things potentially. But to figure that out.

1425
03:33:54.530 --> 03:33:56.329
senate chamber: my view of it is that

1426
03:33:56.480 --> 03:34:03.570
senate chamber: I guess the question for you would be, do you? How much of cognition human, wise, do you think can be explained with

1427
03:34:03.760 --> 03:34:07.099
senate chamber: really good, contextually driven statistical

1428
03:34:07.340 --> 03:34:15.070
senate chamber: learning, which I think is a really powerful mechanism. I think that's the value of, for example, a transformer is.

1429
03:34:15.230 --> 03:34:19.199
senate chamber: the statistics are very contextual, very deeply

1430
03:34:19.390 --> 03:34:28.009
senate chamber: have context down. That's what they do, and enough parameters to give you lots of different kinds of behavior which are still statistical but very context driven.

1431
03:34:28.310 --> 03:34:36.239
senate chamber: So my question to you is, how much of just give me a percent. How much of human cognition is statistical learning, because that's what it is. And I'm not

1432
03:34:36.430 --> 03:34:39.010
senate chamber: saying that statistical learning is lesser.

1433
03:34:39.260 --> 03:34:41.609
senate chamber: I'm just asking the question, like, What's the cut?

1434
03:34:41.950 --> 03:34:46.079
senate chamber: 10%, 50, 80. And then when do we call it human?

1435
03:34:47.890 --> 03:34:51.072
Robert’s iPhone: Yeah, I hope you can hear me.

1436
03:34:51.970 --> 03:35:04.409
Robert’s iPhone: yeah, no, I think that's an excellent question. I'm not sure the answer. It might depend on our ability to build nuclear power plants and build ever larger models. I mean, some people.

1437
03:35:04.920 --> 03:35:10.790
Robert’s iPhone: you know, within the AI community. I mean, obviously, some people who are hardcore argue that eventually it will.

1438
03:35:10.950 --> 03:35:16.510
Robert’s iPhone: It will model all the parts of the brain. It will do everything, but I don't think we can build a

1439
03:35:17.000 --> 03:35:19.460
Robert’s iPhone: a center that big.

1440
03:35:19.780 --> 03:35:23.980
Robert’s iPhone: But maybe I don't know. I think the interesting thing is, though.

1441
03:35:24.420 --> 03:35:31.669
Robert’s iPhone: we don't really study this aspect of human cognition like the cognition we study is this stuff that we bring in in experiments.

1442
03:35:31.900 --> 03:35:32.830
Robert’s iPhone: and

1443
03:35:33.060 --> 03:35:42.989
Robert’s iPhone: we sort of look for deliberate symbolic manipulation. And and things like that. Or we look for reaction times to sort of

1444
03:35:43.190 --> 03:35:45.550
Robert’s iPhone: did something along those lines. And

1445
03:35:47.800 --> 03:35:56.789
Robert’s iPhone: yeah, we, I mean, there are actor models of higher level tasks like expert tasks and things like that, but they're hard to to validate. They're hard to test.

1446
03:35:56.990 --> 03:36:03.890
Robert’s iPhone: But the fact that Actar can do them says something very strong, so

1447
03:36:05.180 --> 03:36:13.890
Robert’s iPhone: I don't know. It's an interesting. It's an interesting question, though, but I do think it it has this underlying sort of bit.

1448
03:36:14.190 --> 03:36:16.739
Robert’s iPhone: Where there's A. There's a division

1449
03:36:17.040 --> 03:36:24.670
Robert’s iPhone: in the in the research community. And there's also a division. And Andrea brought this up. I think it's very important. What are the students thinking.

1450
03:36:24.940 --> 03:36:31.139
Robert’s iPhone: right? I think that's very important, because I think they are. This is really changing how they're thinking about things. But

1451
03:36:31.330 --> 03:36:36.460
Robert’s iPhone: yeah, some people think large language models have nothing to do with humans, and they're offended

1452
03:36:36.600 --> 03:36:38.640
Robert’s iPhone: if you say that other people think

1453
03:36:38.800 --> 03:36:42.419
Robert’s iPhone: there, it's game over. And then there's all the positions in between.

1454
03:36:42.540 --> 03:36:48.710
Robert’s iPhone: So that's just a tricky point. If if we're having, you know, we're having this conversation.

1455
03:36:50.230 --> 03:36:51.900
Robert’s iPhone: and nobody knows the answer.

1456
03:36:53.280 --> 03:36:54.730
Robert’s iPhone: That's my answer.

1457
03:36:55.610 --> 03:36:57.949
senate chamber: Okay, we've got 2. Chris's lined up.

1458
03:36:58.880 --> 03:37:00.100
senate chamber: Chris Meyer.

1459
03:37:02.820 --> 03:37:21.679
senate chamber: Yes, I wanted to take this back to the the question about, are these types of systems, large language models, for example, and other machine learning systems simply repeating patterns. It's seen in its training set? Or is it actually coming up with something new and beyond what it was trained on

1460
03:37:21.860 --> 03:37:26.350
senate chamber: the statistical learning community

1461
03:37:27.660 --> 03:37:37.040
senate chamber: felt slided by, I think, by a lot of reactions to well, it's just doing what it was trained on, so why should we be a bit surprised by it?

1462
03:37:37.180 --> 03:37:48.470
senate chamber: And so a group of folks from Princeton developed a test harness called skill mix, and we know that the statistical learning community is very good at developing test harnesses that show how good that our machine learning system is.

1463
03:37:48.660 --> 03:38:10.430
senate chamber: Skillmix is a test harness that is meant to demonstrate that it's not just a statistical reflection of the capability, but a demonstration of pulling together skills that haven't been trained that the system hasn't been trained on in order to overcome a problem that is posed to it.

1464
03:38:10.630 --> 03:38:20.620
senate chamber: So they presented this to a set of machine learning systems, and it turns out none of them do this except Chat Gpt. 3.

1465
03:38:20.730 --> 03:38:24.969
senate chamber: And this is ancient history. So this was in 2023.

1466
03:38:25.200 --> 03:38:28.250
senate chamber: But now there are calls that chat, gpt.

1467
03:38:28.660 --> 03:38:32.200
senate chamber: chat. Gpt. 3. Isn't actually doing this either.

1468
03:38:32.630 --> 03:38:54.059
senate chamber: So it remains a question in the statistical learning community of is this actually happening? And I think one of the hallmarks of human cognition is this ability to pull together skills in a creative way that it hasn't, that we haven't necessarily seen before, and apply them to solve a problem that is new to us.

1469
03:38:55.640 --> 03:39:02.620
senate chamber: Can I quick add to that? I think part of the reason animals that evolved on earth do. That is because of the

1470
03:39:02.830 --> 03:39:05.389
senate chamber: the efficiency constraints that they're under

1471
03:39:05.740 --> 03:39:10.200
senate chamber: to compute, and it be efficient at the same time as you're developing it.

1472
03:39:10.980 --> 03:39:13.929
senate chamber: That's energy, efficient energy efficient. Thank you.

1473
03:39:14.610 --> 03:39:23.450
senate chamber: Is this a follow on? Or is this? Okay, so follow on and also bring it back to

1474
03:39:24.330 --> 03:39:38.279
senate chamber: the topic of the Symposium, which is hybrid systems and so forth. There is this other big, deep learning systems out there. Alphago and Alphazero are examples of that.

1475
03:39:38.630 --> 03:39:45.319
senate chamber: and they combine deep networks with something called Monte Carlo research.

1476
03:39:45.580 --> 03:39:50.939
senate chamber: which is a form of search which is within the classic AI tradition and

1477
03:39:53.150 --> 03:40:03.609
senate chamber: whether they just regurgitate what's in the training sample alpha 0 could start by just being given the rules of the game.

1478
03:40:03.780 --> 03:40:08.629
senate chamber: and then it plays against itself and came up with

1479
03:40:09.110 --> 03:40:18.499
senate chamber: moves, that after 2,000 years human playing, they said, this, this is new, and nobody thought of this, and this is a God move.

1480
03:40:18.820 --> 03:40:19.840
senate chamber: So

1481
03:40:20.140 --> 03:40:28.269
senate chamber: I think we should give them a little more credit than they're just memorizing all the text on the Internet, and somehow interpolating in between.

1482
03:40:30.610 --> 03:40:56.419
senate chamber: Yeah. So so I definitely agree with Alex there that I mean, there are different classes of system, right? And the deepmind work usually is. It's a class above sort of your standard transformer element in the sense that I view them very much as neurosymbolic architectures, because the kind of Monte Carlo research that they're doing is very much a symbolic problem solving technique.

1483
03:40:56.420 --> 03:41:09.480
senate chamber: But to get back to Chris's point, one sort of methodological point is that whenever there is a new result. Like, for example, they just got very good results on the international math Olympiads.

1484
03:41:09.610 --> 03:41:20.760
senate chamber: They never really tell you. Okay, well, it can do it now much better than it could a few months ago. Why is it? Because there is a new capability in the system?

1485
03:41:20.840 --> 03:41:48.229
senate chamber: What is it? Is it a specific capability for this particular task. Is it a general one? Or did you just collected a whole bunch? More answers to questions, and it can generalize better if they were more transparent with what they're doing to improve their systems. I think that would sort of do a lot to alleviate the kind of cynicism that we see that it's just. You know, it's just opportuning to beat the benchmark.

1486
03:41:49.360 --> 03:41:57.659
senate chamber: This follow on or new comment. This is definitely a follow-on. Follow on to Alex's. Follow up.

1487
03:41:57.760 --> 03:42:08.740
senate chamber: Alpha 0 is claimed to have played more games than humans have all humans have in the history of human gameplay.

1488
03:42:11.010 --> 03:42:20.040
senate chamber: And I'll add to that comments. It also didn't have culture and generations. That's why humans were frozen in only a small subset of the problem space.

1489
03:42:20.410 --> 03:42:34.719
senate chamber: If you would have trained Alphago so that it just could teach itself over and over generationally, it may not have generated something new, but maybe it would. I mean, I'm not that surprised that through reinforcement, learning, doing millions and millions of trials, it came up with a new part of the problem space. It's fine.

1490
03:42:34.940 --> 03:42:38.880
senate chamber: and we should give him credit, but it's reinforcement. Learning, I think, is what you would get.

1491
03:42:42.150 --> 03:42:44.459
senate chamber: Are you ready, Chris? Nancy?

1492
03:42:45.300 --> 03:42:48.550
Christopher Dancy: Sure there were so many. Oh, man, I don't

1493
03:42:49.340 --> 03:42:51.289
Christopher Dancy: like. I'm wondering and talk about some of.

1494
03:42:51.740 --> 03:42:52.190
senate chamber: Okay.

1495
03:42:52.190 --> 03:42:52.875
Christopher Dancy: Those?

1496
03:42:54.820 --> 03:43:10.292
Christopher Dancy: because, yeah, there's there's questions of, I mean, even that point of like, I've never seen this movie before. There's questions of, does that mean it didn't exist right? Has nobody ever done it? I don't think necessarily that's the case. It might just be that it was never recognized, and so I think this point is somewhat related. Actually,

1497
03:43:11.310 --> 03:43:13.949
Christopher Dancy: and some of the conversation we were having.

1498
03:43:14.540 --> 03:43:21.130
Christopher Dancy: and in thinking about Lm, somewhat as behavioral, this? I think.

1499
03:43:21.670 --> 03:43:36.510
Christopher Dancy: not understanding. And this is kind of to Jan's point as well, not saying limitations of the models in terms of representation can be really problematic. And this is a problem that we have in cognitive science writ large right in terms of who's represented in our data and and the ways in which

1500
03:43:36.580 --> 03:44:05.489
Christopher Dancy: differences between communities may be represented there, and what that might mean for our results, and the data that we have, and certainly the data felt that fed into Centaur, for example. But this matters a lot with Llms partially because of some of the things we've been discussing, which is the scale and ease of use of those systems. And now the change in the way that Cobb neural grad students are thinking that way. That's really scary, I'll admit, like I'm not surprised. My Cs students might think that way sometimes, but

1501
03:44:06.790 --> 03:44:07.710
Christopher Dancy: but

1502
03:44:07.910 --> 03:44:35.840
Christopher Dancy: so that that idea of being able to scale and use them in these ways and and how it's changing the way people think again, think about it as a knowledge source almost really matters. And so I think we certainly have to really, be careful about that as community as we talk about it, and be weary of those issues that undergird large language models which there's full swaps not full. There's plenty of people who have been ringing the bell for problems with them as

1503
03:44:35.860 --> 03:44:49.019
Christopher Dancy: sequence generators they are, and ways that we are kind of going back to category errors. Think about them wrong overall. And I think the other point that I wanted to make, because we've been talking about generative models, but

1504
03:44:49.090 --> 03:44:56.159
Christopher Dancy: we have to be careful also to conflate different models. So like Chat Gpt is very different from an anthropic from a llama

1505
03:44:56.250 --> 03:45:01.220
Christopher Dancy: from a Math Neo, which is one of the few open

1506
03:45:01.400 --> 03:45:24.510
Christopher Dancy: model. Open weight. Open data models exist, and we have to be careful because they're not only different sizes, but some of them we had no idea the way they work, or when they change at all right, like from Chat Gpt in between. Dan's testing could have changed in some way. We would know, because we don't, because it's not open, despite the fact that the company's name is Open AI. And so I think for me. It's been really interesting watching

1507
03:45:24.860 --> 03:45:28.720
Christopher Dancy: scientific use of some of these systems, particularly because

1508
03:45:29.150 --> 03:45:44.570
Christopher Dancy: there in the past systems that have been closed and cost money weren't used in these same ways. And we're like using chat gpt in that way. And part of it is because of, you know, popular culture. But I think, as a community, we have to be very careful with

1509
03:45:44.570 --> 03:46:05.040
Christopher Dancy: the ways in which we engage those systems, not only given the the scientific issue of some of those things right? The philosophical issue, some of those things, but also back to the environmental impact. Right? It's it's not nothing to use those models, and there there are actual implications to using them often. So that's just my my point. And as we think about engaging with these models, and being

1510
03:46:05.080 --> 03:46:08.129
Christopher Dancy: very careful in the way we engage in being principled in it.

1511
03:46:13.250 --> 03:46:15.279
senate chamber: Okay, now we got any other

1512
03:46:15.390 --> 03:46:18.839
senate chamber: comments on that, or, Oh, Mike, I heard from you.

1513
03:46:19.600 --> 03:46:25.600
senate chamber: So I kind of want to go back to some of the stuff that Christian said in the introductory part of the session.

1514
03:46:25.910 --> 03:46:47.869
senate chamber: because I think I think, while it's loads and loads and loads of fun to sit here and just slag Llms over and over again. I do think that there are some opportunities being created here that are things that we might be interested in leveraging if we could figure out how to do it like one of the things that we've always had. This challenge with in Act R is is, you know you're you're

1515
03:46:48.140 --> 03:46:58.029
senate chamber: what does a college sophomore know? Right? When it walks into the experiment? You have to figure out how to model some of the knowledge that they have that they might be bringing to bear on the task that you're giving them.

1516
03:46:58.420 --> 03:47:17.029
senate chamber: And it seems like, if they're not useful for anything else. They may be useful for helping us solve some of the problems of what sort of a reasonably generic set of procedural and a little bit of declarative knowledge that we could maybe assume that the average person has to some degree that we don't have to hand engineer

1517
03:47:17.050 --> 03:47:40.920
senate chamber: for each one of our act, our models. I think those kinds of opportunities. There's a real morass of what goes on. If we start worrying about how this thing operates as is it a cognitive theory? Blah blah blah. But maybe these things have some utility to us in ways that we can leverage to make what we're doing better or easier or more efficient, and I want to see if I can nudge

1518
03:47:41.120 --> 03:47:46.520
senate chamber: the conversation towards ways in which it might actually these things might actually be helpful to us.

1519
03:47:47.290 --> 03:48:02.929
senate chamber: I'd like to thank the speaker for asking a question where the answer is on the screen. That's in partial compensation for the extra drink token. Shi Yu Wu working with people from Bosch.

1520
03:48:03.010 --> 03:48:21.239
senate chamber: one of Christian's postdocs, Alexandro. They're taking at the top, you see, essentially a description of that model. It's going to spit out a trace, and the trace is going to be spat into an Llm. And the Llm. Is going to be asked to give advice on how to optimize a factory

1521
03:48:21.460 --> 03:48:29.470
senate chamber: and to get they claim. Well, I guess we claim it gives better advice and more explainable advice because of the trace that's been injected into it.

1522
03:48:34.350 --> 03:48:37.800
senate chamber: So I return you to your regularly scheduled bonfire.

1523
03:48:39.630 --> 03:48:43.379
senate chamber: Andrea has his hand up online. Andrea.

1524
03:48:44.550 --> 03:48:45.563
Andrea Stocco: Thank you.

1525
03:48:47.000 --> 03:48:52.660
Andrea Stocco: I don't know if what I'm going to say is gonna make much sense. But there are 2 different

1526
03:48:52.850 --> 03:49:06.469
Andrea Stocco: aspects to Neosymbolic right? One is the fact that you know you can integrate the system actar with the systems, and you can harness their extremely appealing power, which is where I think most of the discussion is going.

1527
03:49:06.600 --> 03:49:18.550
Andrea Stocco: and that is really important, for well, I guess, for the thing of the next panel, like Atari as a software. And what's the direction to pursue? But there's the other aspect of like, what do these systems can tell us about the theory?

1528
03:49:19.290 --> 03:49:27.499
Andrea Stocco: And and I think that this is this speaks more, maybe, at least from my like

1529
03:49:27.650 --> 03:49:33.859
Andrea Stocco: work as a my other heart as a neuroscientist. Most of the work that May and Kathy have been doing.

1530
03:49:34.090 --> 03:49:48.499
Andrea Stocco: and I'm wondering whether the panelists can talk more about like, not only how can we use these tools? Because, of course, we all want to use them, but also like, what can we learn from this integration approach that could be helpful for

1531
03:49:48.910 --> 03:49:54.830
Andrea Stocco: for the community and for the architecture and the theory, and I think that

1532
03:49:55.080 --> 03:50:00.929
Andrea Stocco: intuitively there should be probably a connection between these and the problems that were raised in the very 1st panel.

1533
03:50:15.490 --> 03:50:17.389
senate chamber: Did they kill the debate?

1534
03:50:19.960 --> 03:50:23.049
senate chamber: No, they're looking to see who who gets to do it

1535
03:50:31.510 --> 03:50:41.190
senate chamber: like it's the-. The integration is always a bit tricky, alright

1536
03:50:41.990 --> 03:50:47.950
senate chamber: like. I sometimes worry about large language models having kind of baked in kind of tendencies that might be.

1537
03:50:48.080 --> 03:50:55.870
senate chamber: not what we want from our particular cognitive model or architecture that we've been trying to construct.

1538
03:50:57.910 --> 03:50:58.610
senate chamber: Wish

1539
03:50:59.390 --> 03:51:07.820
senate chamber: is why, I guess I lean more towards trying to create kind of knowledge systems that combine the 2. But that is me. And then down the road

1540
03:51:08.300 --> 03:51:14.970
senate chamber: do a bunch of training myself when I don't have the resources that Google or Microsoft have?

1541
03:51:17.570 --> 03:51:24.919
senate chamber: When I was thinking about the theory of mind problem, I do think that theory of mind is something that potentially really benefit from

1542
03:51:25.030 --> 03:51:33.710
senate chamber: maybe like something like an actor architecture, or conventional architecture that goes through some sort of set of reasoning steps about people in a situation.

1543
03:51:34.050 --> 03:51:37.480
senate chamber: Large language model gives them sort of the more fuzzy kind of

1544
03:51:37.660 --> 03:51:48.079
senate chamber: predictive knowledge that useful. For, like the situation where, like where you've seen like this person over a period of time like a 2 h interview, so you can get that vibe of them.

1545
03:51:49.860 --> 03:51:51.879
senate chamber: But I guess I don't.

1546
03:51:53.550 --> 03:51:57.100
senate chamber: The set of strong answers to.

1547
03:51:57.800 --> 03:51:59.219
senate chamber: We can solve these problems.

1548
03:52:00.910 --> 03:52:05.329
Andrea Stocco: Just to clarify. I wasn't just restricting it to Llms.

1549
03:52:05.780 --> 03:52:10.460
Andrea Stocco: Any kind of neurosymbolic approach to me is interesting.

1550
03:52:13.040 --> 03:52:25.259
senate chamber: I was. Gonna say, I also don't have a good answer to this, but I think to Christian's point, that the big advantage of the kind of symbolic structural models are that we know what they're doing. And we lose that

1551
03:52:25.430 --> 03:52:34.580
senate chamber: when we hand things over to things that are being generated, particularly by algorithms that we don't totally know what they're doing and can change on us at various points.

1552
03:52:35.070 --> 03:52:42.080
senate chamber: So I think there's a lot of benefit to trying to incorporate them in small ways, as has been suggested.

1553
03:52:42.740 --> 03:52:48.830
senate chamber: several in several variations, but that the kind of theoretical models

1554
03:52:49.190 --> 03:52:55.670
senate chamber: need to be more in a steering position so that we don't lose the kind of explanatory power that they're

1555
03:52:55.830 --> 03:52:57.340
senate chamber: to provide for us.

1556
03:52:58.440 --> 03:53:13.599
senate chamber: 1 1 quick thing I'll add on, that is so. So what obviously, what people have explored is the the chain of thought output in reasoning models. As a way of introspecting in those models, it turns out that

1557
03:53:13.600 --> 03:53:36.210
senate chamber: it does not necessarily truthfully reflect on what they're actually doing. It's just. And in that way they may be human, like in the sense that they're just putting out what they've been trained to put out, not actually what they're doing. If it were possible to somehow train these models to be somehow truthful about putting what they're doing

1558
03:53:36.210 --> 03:53:47.100
senate chamber: through some form of training or algorithmic constraints, or whatever that I think that would be a major advance in terms of making them more introspectable.

1559
03:53:47.990 --> 03:53:50.530
senate chamber: So so it has gone to 15.

1560
03:53:50.780 --> 03:53:58.860
senate chamber: It has gone to 15, unless the next chair or the workshop chair comes after me.

1561
03:53:59.680 --> 03:54:06.849
senate chamber: We need to declare the session over the schedule is incomplete and doesn't say when we come back.

1562
03:54:07.720 --> 03:54:16.009
senate chamber: But I'll quickly at in 15 min, 2, 32, 30, okay, and at 2 30, approximately, there's a

1563
03:54:16.450 --> 03:54:18.870
senate chamber: car headed to the airport.

1564
03:54:19.390 --> 03:54:32.990
senate chamber: I hope that there are a few people in the car, because it's going to be a great session to wrap up the workshop right? I'd love to stay. But Delta and time does not wait for any single person.

1565
03:54:34.810 --> 03:54:39.830
senate chamber: Kristen, I'm going to the bar at 2 30. If anybody wants to join me.

1566
03:54:47.700 --> 03:54:48.710
Christopher Dancy: Thanks. Everyone.

1567
03:54:53.430 --> 03:54:58.520
senate chamber: Is that it sneak up

1568
03:54:58.760 --> 03:55:07.180
senate chamber: to to make up on time? Let's introduce our future of actor panel. Andrea. Why don't you just take it away.

1569
03:55:09.031 --> 03:55:14.949
Andrea Stocco: Sorry I really didn't hear what you just said. I know this is slowing us down. Could you repeat.

1570
03:55:14.950 --> 03:55:19.660
senate chamber: Sure. I just said, it is time for you to start.

1571
03:55:20.020 --> 03:55:33.600
Andrea Stocco: All right. Perfect. Okay? So then, I will not waste any time. So thanks everybody for making it to the end of the day. To this workshop, which I think is going to be a very fairly interesting end of end of the day panel.

1572
03:55:33.890 --> 03:55:42.429
Andrea Stocco: And this is going to be a different, somewhat different panel and traditional future of Acta. And there are reasons for this that I hope you're going to see in the next few

1573
03:55:42.590 --> 03:55:56.409
Andrea Stocco: minutes. I'm gonna introduce the reason for this panel in this very 1st set of slides. But then I think that most of the attention should go to what the panelists actually are going to say not to what I am saying right now.

1574
03:55:57.140 --> 03:56:24.909
Andrea Stocco: So let me get started. So the great thing about coming to these workshops is that you get to see all the ways in which Aktar is used and all the things that Aktar represents to different people. I like to think a lot in terms of the theory. But of course, Aktar is not only a theory, it's also the most successful cognitive architecture, a great community, a software that can be used. And I'm going to focus a little bit in this slide on these last 3 points.

1575
03:56:25.180 --> 03:56:51.149
Andrea Stocco: And because these points prompted us in the past month to do some considerations. The 1st one is that Akta, compared to its peer architectures, has a much larger and more diverse user base than anyone we know, or we could survey. For instance, in 2020, the users of Actar counted by the number of papers in which it was used, including people that never show up in this community that we don't really know really

1576
03:56:52.030 --> 03:56:56.750
Andrea Stocco: were bigger than the other next 3 architectures combined.

1577
03:56:56.860 --> 03:57:08.609
Andrea Stocco: And just to give you an infographic that Dan Bodel Genty shared with us. This is a map of all the locations in which the Actara software was downloaded in 2024 from the website.

1578
03:57:09.500 --> 03:57:21.540
Andrea Stocco: and to refer back to a paper that many of us have cited over the years. This is the combined use combined output in 2020, when the survey was made. So you can see that basically, there is no comparison.

1579
03:57:22.320 --> 03:57:42.269
Andrea Stocco: And this puts Akta on a strange situation because Akta's software ecosystem support is incredibly efficient. Nobody can complain about that, but it's also incredibly lean and frugal. It is supported entirely by the teams at Cmu, and this support, as far as I understand, is entirely generated by research grants.

1580
03:57:42.500 --> 03:57:55.469
Andrea Stocco: We don't pay for Akta. We always assume that Akta is free, and then we can bother Dan anytime we need, and we do, and then somehow always find time to respond within a few milliseconds of the email being sent.

1581
03:57:55.660 --> 03:58:12.800
Andrea Stocco: But we shouldn't take it for granted, because this current situation, being frugal, lean, and efficient, is also like a little bit paradise. So the actor use is much, much bigger than what akta support is like, and to be fair, this is not to scale, because on the right hand side, in practice you have a very big rock.

1582
03:58:14.070 --> 03:58:20.680
Andrea Stocco: but at the same time, we need to think about making this balancing act as sustainable in the long term.

1583
03:58:21.780 --> 03:58:29.250
Andrea Stocco: because the user base is growing. When I joined the tech community almost no more than 2 decades ago. At this point

1584
03:58:29.370 --> 03:58:41.649
Andrea Stocco: the people who were using it were pretty much all the people that you can count at the Aktar workshop and at the Cogsai gatherings. And now we really see like Actar papers popping up everywhere.

1585
03:58:41.800 --> 03:58:45.890
Andrea Stocco: At the same time. The research funding in the Us. Is dwindling.

1586
03:58:46.190 --> 03:58:53.809
Andrea Stocco: and the software and application landscape is changing. And I think that this is something that we've been hitting on and on for 3 panels in a row.

1587
03:58:54.260 --> 03:59:15.840
Andrea Stocco: which means that the long-term support of the architecture requires some rethinking, and this rethinking, there are many ways in which we can be planning ahead. But at least these 4 points need to be addressed, I think an integration with a broader set of tools and applications, more robust pipelines. This is something that Christopher Nancy brought up before, and I really appreciate it.

1588
03:59:15.840 --> 03:59:24.719
Andrea Stocco: easy ability to scale and combine models. And of course this would also allow us to allow to attract users and funding sources from other domains.

1589
03:59:24.810 --> 03:59:42.220
Andrea Stocco: and to summarize something that has been clearly voiced over in the previous panel. The current situation is, this. Actar was born out of the psychology domain, but everybody seems to be looking up at other domains and AI and logical models

1590
03:59:42.320 --> 03:59:50.680
Andrea Stocco: in particular. Now I know that I always put memes in my talks, but I would like to point out that the author of this meme is actually Christian Levier.

1591
03:59:51.416 --> 03:59:52.153
senate chamber: Yes.

1592
03:59:54.430 --> 04:00:13.320
Andrea Stocco: All right. So this brings the question, how do we make Aktar self-sustaining? A few months ago we started thinking about this. I was involved actually in these discussions with a bunch of other folks that I'm going to name in a few slides. And we started by looking at like Aktar purely as a software, and why it exists.

1593
04:00:13.500 --> 04:00:42.769
Andrea Stocco: And if you want to look at it from the point of view of like hypothetical funding agency, that we would be asking money for like support Actar, they would ask us, what is Aktar and Actar would be probably roughly described like this, a particular type of software that supports application in the scientific computing domain. So it lives in this kind of red ecosystem is in the open source space. It's not a language like Python or R, but it's also not a product like a Microsoft office.

1594
04:00:42.880 --> 04:00:48.930
Andrea Stocco: And there are several interesting domain. Interesting examples in this kind of like redbubble domain.

1595
04:00:49.400 --> 04:01:00.730
Andrea Stocco: I decided to collect a few, just to show you an idea of the variety of the space, and there are many different axes and dimensions in which you can split this giant bubble. So I decided to focus on 3.

1596
04:01:00.950 --> 04:01:04.020
Andrea Stocco: How is the development of the project led?

1597
04:01:04.420 --> 04:01:16.519
Andrea Stocco: What is the process by which development occurs? Is it like a bottom up or top down situation? And how is the project supported? At least there are some of the common

1598
04:01:16.970 --> 04:01:30.499
Andrea Stocco: options that occur in this domain. But to give you an idea of the variety, I put like a set of 6 examples, and I think that most of us are familiar, if not with all of them, at least with like half of them.

1599
04:01:31.420 --> 04:01:50.159
Andrea Stocco: You see that, for example, at the top level, you have probably the most common thing, open source software. Everybody's familiar with which is the Python language, which was historically led by a single person, Ross, on the original development. And I think this has changed. Recently it was nicknamed the Benevolent Dictator for Life.

1600
04:01:50.260 --> 04:01:59.799
Andrea Stocco: It has switched to a bottom-up development where people actually suggest and propose improvements to the language, and it has a big support, essentially from industry partners.

1601
04:02:00.180 --> 04:02:23.659
Andrea Stocco: Then you can go down the scale. There are some interesting like new imaging software. Scipy is a package for Python R is the standard language of statistical computing. You see, like slightly different changes in who leads? Who develops? How is the support. Granted. Some people are lucky. Like the team. Cal Friston's team behind Spm has received 30 years of support and interactive from the Biocom Trust.

1602
04:02:24.060 --> 04:02:46.499
Andrea Stocco: a foundation in the Uk Cypy is supported by individual donation and industry partners. Psych toolbox was originally completely free and led by people who were working out of the goodness of their heart. This has lasted for 30 years, but now they have recently decided to switch to a licensing scheme. The software is still free. But you actually have to license to use it legally.

1603
04:02:46.500 --> 04:03:02.009
Andrea Stocco: Now, everybody has colleagues that work in somehow this space. And you can find examples all around. I have a colleague who's actually like in the office next to mine, who is Aria Rockham. And I want to put as an example of a Microsoft community

1604
04:03:03.020 --> 04:03:15.830
Andrea Stocco: is software, so that he has developed is called pi Afq for automatic fiber tracing. And it's something that has a very niche use in the domain of neuroimaging. It does diffusion, sensor, imaging analysis.

1605
04:03:16.040 --> 04:03:37.919
Andrea Stocco: Now, the thing that is shocking. And I talked to Ariel about in the past few days. I interview it preparing the slide is how much time the development of the software cost. It works about on the software side, just on coding about 8 HA week. He has a full graduate student that works only on this project. One of his previous students described the lab as essentially a software startup

1606
04:03:37.920 --> 04:03:47.889
Andrea Stocco: it has received, supporting from nih and DOE, but this support is also finishing. It has an active ecosystem. It is entirely based on github. People track bugs, request features to Github.

1607
04:03:48.030 --> 04:03:57.539
Andrea Stocco: and he does an enormous amount of work going around and traveling the country, spreading how to use the software and how to implement projects with the Pi. Fq.

1608
04:03:57.740 --> 04:04:09.120
Andrea Stocco: And because support is windy. Right now, he's also exploring, like essentially developing a private company that will just sell tutorials for the software and licensing.

1609
04:04:09.390 --> 04:04:17.339
Andrea Stocco: Now, Acta is somewhat should be in this list, but we don't have really

1610
04:04:17.440 --> 04:04:27.109
Andrea Stocco: something to put there yet in. Well, we do have something to put there already, but if we want to rethink, we need to put some kind of values to this

1611
04:04:27.230 --> 04:04:30.600
Andrea Stocco: entries. They put in this small list

1612
04:04:31.050 --> 04:04:41.129
Andrea Stocco: who is going to be like the development team behind next generation. Rata, how is it going to be growing? How is it going to get supporting in the next 10 years? 20 years, and so on.

1613
04:04:41.330 --> 04:04:42.840
Andrea Stocco: Well, thankfully.

1614
04:04:43.450 --> 04:04:51.270
Andrea Stocco: The National Science Foundation has a project, a program that is entirely dedicated to supporting this type of transitions.

1615
04:04:51.530 --> 04:04:55.089
Andrea Stocco: We discovered it about 7 months ago.

1616
04:04:55.310 --> 04:04:58.110
Andrea Stocco: The existing of this particular program, called Pause.

1617
04:04:58.290 --> 04:05:25.390
Andrea Stocco: It has divided in it is divided in 2 phases. The 1st phase is what they call ecosystem discovery. And the idea is like, okay, try to collect data about what is the possible use of your software application and try to find out how it can grow. And phase 2 actually is like a significant grant that helps researchers transition from like their own lab, grown software to something that is self-sustaining in the future.

1618
04:05:25.850 --> 04:05:55.199
Andrea Stocco: And this is the the general vision of the program, like the idea, is to break this kind of like scientist developer software and share with other external users into something that is like a more comprehensive system that can be used for that includes, like richer documentation process to onboard and train people for development, a bigger community tools for pipelining and scaling the system and concerns about security, privacy, any kind of any kind of possible

1619
04:05:55.200 --> 04:05:59.930
Andrea Stocco: problems, legal and ethical that can arise from misuse in different domains.

1620
04:06:00.470 --> 04:06:06.229
Andrea Stocco: So we did the normal thing that anybody will do in this case, like, Hey, this program exists.

1621
04:06:06.860 --> 04:06:18.219
Andrea Stocco: we should apply, and I have to thank everybody who contribute to this, but especially Christian, who drafted an entire 1st draft of the proposal in what I think was like 2 days.

1622
04:06:18.390 --> 04:06:38.139
Andrea Stocco: and we submitted and kept the fingers crossed, and even like, in the crash of funding from Nsf. We got the surprising news that we got great reviews than we were recommending for funding. This hasn't translated into actual funding. So I'm putting that henceforward there is a big gap between the 2. But if everything goes well.

1623
04:06:38.140 --> 04:06:48.930
Andrea Stocco: the grant will give one year to do the discovery part, basically interviewing people and collecting information. Now also to put their hands

1624
04:06:48.970 --> 04:07:00.940
Andrea Stocco: forward about this doesn't mean that we're assuming or claiming any significant role. We're not saying like, Oh, we're going to take over Actar. We are going to be the real team. We're just here to collect data and propose.

1625
04:07:01.430 --> 04:07:26.029
Andrea Stocco: So there is a roadmap at this very point, and the main point of my introductory slides is that the roadmap involves significant discussions with all of you starting with this community and the people that are here at this workshop because we need to collect important information about how you use actar. What is your vision for actar? What are the roadblocks that you're experiencing in using actar, or how you would like to use actar better for your own projects.

1626
04:07:26.160 --> 04:07:53.160
Andrea Stocco: but we also plan to reach out to other people that you may know, and you could put us in touch to researchers who might be interesting to use Aktar. But they really don't use actar. I sometimes go on committees with colleagues in computer science and engineering, and they know something about Aktar, but they really never made the 1st move to actually learn how to use it, or they would like to have something, but maybe the students are scared, because, like well, I don't know how to integrate this with like some kind of sci-fi architecture, and so on

1627
04:07:53.690 --> 04:07:58.069
Andrea Stocco: and being students, they don't have often the time to read papers, and so on.

1628
04:07:58.520 --> 04:08:17.169
Andrea Stocco: There are also the researchers that we would like to contact people like Arya Roke and my colleague people who do have developed some software, and they've gone a different pathway than actor has gone, and we would like to understand the pros and cons of their pathways. So we're going to reach out. And if you have any suggestions on any of these points, you don't even need to wait for us. Please let us know.

1629
04:08:17.440 --> 04:08:34.910
Andrea Stocco: So to briefly summarize the goal of this kind of like introductory talk to this panel. Akta is a great history because Akta was an open source development platform for a theory before these things actually were common name places.

1630
04:08:35.320 --> 04:08:47.139
Andrea Stocco: And of course, like, from my point of view. This is like exactly like what a great theory should be. It should be a theory, but it should be computation. It should be implemented in a software, it should be open source so that people can use it and grow.

1631
04:08:47.270 --> 04:08:56.840
Andrea Stocco: The future of Actara is to reach the next level of joy and pleasure and satisfaction which is making it a self-sustaining ecosystem.

1632
04:08:57.880 --> 04:09:20.439
Andrea Stocco: And of course, I imagine that people have different list of things that they would like to see achieve in the next decade or so. I have my own personal vision, and I'm putting it. There is just to give you an idea. I have actually thought about it. I'm going to delete the slide immediately, because I don't want to bias where it's going, and instead, I'm going to introduce the panelists.

1633
04:09:20.540 --> 04:09:43.719
Andrea Stocco: Thanks to Christian. We put together an incredible group of people that actually talk much better than I about what are the possible futures of Akta, and they include 7 different uses and views of how Akta should be used. There are people that come from different fields in which Akta is applied, 2 professors, 2 researchers. One is actually both a professor and an entrepreneur.

1634
04:09:43.970 --> 04:10:00.660
Andrea Stocco: and Wang is, of course, the person that has been the future of Akhtar, and will be the future of Akhtar until the foreseeable future, which is the one and only dang. And with this I actually relinquish my time, and I'm ready to start to hear from the panel.

1635
04:10:08.780 --> 04:10:16.689
senate chamber: Okay. I'd like to invite all the panel members to come up while they're getting ready. Did we want to take one question?

1636
04:10:19.600 --> 04:10:20.430
senate chamber: Alright?

1637
04:10:23.270 --> 04:10:24.970
senate chamber: I guess.

1638
04:10:25.220 --> 04:10:34.249
senate chamber: All right, I guess my one question is so. The Aptar that I've generally used is the pipeline implementation of Aptar that was developed at Carlton University, where I'm at

1639
04:10:34.700 --> 04:10:58.380
senate chamber: is that something that's like is there interest, I guess, in maintaining alternate code base implementations of actar for the interest of accessibility? I mean, I love lisp. I learned lisp in undergrad, but I am not currently teaching my students lisp.

1640
04:11:04.940 --> 04:11:05.680
Andrea Stocco: Awesome.

1641
04:11:06.450 --> 04:11:09.200
Andrea Stocco: I think this is one of the questions.

1642
04:11:09.200 --> 04:11:26.989
senate chamber: Point of view. We've had sort of the dance reference implementation that he's generalized from list to python with a wrap, but another at least half a dozen by my count. Probably more implementations have popped up, but

1643
04:11:27.200 --> 04:11:45.159
senate chamber: for different uses, like, for example, Dario has his supermodel implementation. It's more oriented to Hci Greg for robotics, etcetera, etcetera, and I think part of managing the ecosystem would be to to figure out a way of

1644
04:11:45.370 --> 04:11:59.260
senate chamber: whether sort of maintaining coherence there, or alleviated the need for alternative implementation. Again, it's it's wide open. But I think that's definitely an issue that that needs to be addressed as part of the ecosystem.

1645
04:12:04.660 --> 04:12:12.989
senate chamber: Okay, I think to get things started. I liked the alphabetical thing we did before, and then take a question while we transition. Sorry, Dan.

1646
04:12:13.100 --> 04:12:18.290
senate chamber: But let's lead off with Dan 5 min.

1647
04:12:18.820 --> 04:12:24.350
senate chamber: I don't need 5 min. I don't have any prepared slides or anything. I guess

1648
04:12:24.750 --> 04:12:28.959
senate chamber: I'm just gonna propose a question to put out discussion out there.

1649
04:12:31.190 --> 04:12:36.639
senate chamber: There is only one, Dan, so I guess there's a

1650
04:12:36.800 --> 04:12:43.010
senate chamber: an issue for this whole project, right? And how do we kind of make more dance.

1651
04:12:44.110 --> 04:12:48.730
senate chamber: or perhaps distribute that knowledge in a way that it can be self-supporting.

1652
04:12:48.930 --> 04:12:53.880
senate chamber: and I don't know the answer to that. So I guess I'm just going to put out a question for discussion.

1653
04:12:56.220 --> 04:12:57.170
senate chamber: Can we respond?

1654
04:12:57.780 --> 04:13:00.497
senate chamber: Large language models are the answer.

1655
04:13:01.250 --> 04:13:03.330
senate chamber: yeah, we're calling them back to dance.

1656
04:13:06.370 --> 04:13:13.730
senate chamber: Yeah, can I briefly mention that this path has been taken by another company of architecture soar.

1657
04:13:14.310 --> 04:13:22.750
senate chamber: they do have organization or company is development picture.

1658
04:13:23.310 --> 04:13:26.940
senate chamber: That's their main business.

1659
04:13:36.640 --> 04:13:39.215
Andrea Stocco: If you can interject

1660
04:13:40.720 --> 04:13:51.470
Andrea Stocco: the the issue that then Rose is actually one of the issues. Specifically, I described in the post proposal, and that I think the term that they use is onboarding.

1661
04:13:52.380 --> 04:13:59.920
Andrea Stocco: And historically, like every kind of like scientific software starts with one developer or a couple of developers.

1662
04:14:00.260 --> 04:14:13.709
Andrea Stocco: But eventually part of the support is actually the process by which new people get to recover the historical knowledge and get to learn how to develop, continue developing and maintaining the software.

1663
04:14:13.970 --> 04:14:15.986
Andrea Stocco: And we don't have.

1664
04:14:16.950 --> 04:14:25.679
Andrea Stocco: we're still, this is one of the list of priorities that we have. And actually, we're really interested. If anybody has any experience with this.

1665
04:14:31.560 --> 04:14:35.360
senate chamber: Do you have any more questions or responses to Dan's initial question?

1666
04:14:40.440 --> 04:14:49.800
senate chamber: not to to become another dam. But when we were working with Alex on the graphic module and making some modification and things like that.

1667
04:14:49.920 --> 04:14:58.060
senate chamber: We interacted with you. We kind of also like, went into the code and tried to make some modification to some extent.

1668
04:14:58.490 --> 04:15:00.833
senate chamber: isn't working properly. But

1669
04:15:01.880 --> 04:15:11.619
senate chamber: Andrea was proposing some academies, and we have the summer school. Could there be a summer school that is much more oriented to

1670
04:15:11.770 --> 04:15:18.330
senate chamber: going in depth into module books and stuff like that, and I would be very interested.

1671
04:15:22.730 --> 04:15:30.889
senate chamber: I guess I guess the question would be, how many people would we get at something like that

1672
04:15:32.160 --> 04:15:33.640
senate chamber: cause? I guess

1673
04:15:34.000 --> 04:15:39.660
senate chamber: most of our user base are not the programmers that are building it. They're the people that are using it as a tool.

1674
04:15:39.760 --> 04:15:45.219
senate chamber: So I don't know where we find the group of people that would overlap. With that.

1675
04:15:46.580 --> 04:15:48.950
senate chamber: I guess I kind of fell into my role.

1676
04:15:49.640 --> 04:15:51.210
senate chamber: I've been there for a while.

1677
04:15:55.330 --> 04:15:57.770
senate chamber: If I can jump in with a related point

1678
04:15:57.810 --> 04:16:17.360
senate chamber: part of the complexity here as opposed to sort of traditional software environments that were mentioned like Linux, and all that is that there's at least 4 potential kind of contributions to the Aktar sort of ecosystem. So one is sort of the architecture itself.

1679
04:16:17.360 --> 04:16:42.879
senate chamber: and that can be fairly tricky, I think, to modify. I mean, it's like hacking the Linux kernel right? The second one is contributing new modules to the architecture, and you know there's been an infrastructure in place to do that for like 20 years, which I think has been moderately successful. It should be, you know, nice modular, except the difference between a software module and architectural module can be subtle, and there can be dependencies.

1680
04:16:43.070 --> 04:16:56.189
senate chamber: The 3rd one is models, and you know, people contribute models. But by and large, you know, people look at it. It's not like. They're building their models on top of models which would imply, like some more validation there

1681
04:16:56.190 --> 04:17:19.480
senate chamber: and then. Finally, the whole world of applications, right various utilities around the architecture, things to connect to frameworks, and all that. That's 4 very different kind of contributions, with very different thresholds and types and verification, and all that as kind of issues that you get into it when you have an open ecosystem.

1682
04:17:19.530 --> 04:17:31.050
senate chamber: and that there's a lot of complexity there that's really not trivial. And I think it's a pretty unique set of issues. From the point of view of software ecosystems.

1683
04:17:33.300 --> 04:17:39.829
senate chamber: David, you made a point about Linux Kernel development as well. Did you want to expand upon that before we move on?

1684
04:17:50.310 --> 04:17:52.030
senate chamber: Okay, I guess we'll move on, Mike.

1685
04:17:58.280 --> 04:17:59.240
senate chamber: Try to share.

1686
04:18:04.920 --> 04:18:08.439
senate chamber: I have slides because there's no way I can speak for only 5 min without structure.

1687
04:18:10.360 --> 04:18:11.180
senate chamber: Oh.

1688
04:18:11.260 --> 04:18:27.250
senate chamber: all right. So I have 2 sort of perspectives coming into this one of which I think, is reasonably not unusual. I'm a researcher. I use actar. I do it for hci oriented workout neuroscience work. So that puts me in a slightly different camp than some of the other people.

1689
04:18:27.250 --> 04:18:45.379
senate chamber: and I'm a developer of Actr. I wrote the modular system and discrete event simulator on which act R is based. I didn't know it was called the Discrete Event Simulator at the time, but that's what it is. I still have right access to the act. R subversion, repository. How many people have right access?

1690
04:18:46.020 --> 04:19:07.189
senate chamber: Yeah, not many. Yeah, 3 or 4. The coolest thing that I wrote that was none of those pieces. But it was a thing that if you implemented a little graphical user interface in Macintosh Common List Actar could see it and click on it and type to it. And you didn't have to know how to make that visible like. There was a little piece of software layer that did that right. So you'd have a window

1691
04:19:07.340 --> 04:19:21.100
senate chamber: oops wrong button. You get a window, and you could see it right. That's incidentally my lasting contribution to science is a little red ring. Every time some non-actar person has a simulation with a little red ring on it. I want to go up and take credit for it.

1692
04:19:21.130 --> 04:19:50.720
senate chamber: but things are different, although some things are less different than I would think right. So the base architecture, right? The main core of the theory changed a lot from the early 19 nineties through the mid 2 thousands. Right? We went through act R. 2, 3, 4, 5, 6, all from 1991 to 2,005 ish like that was pretty regularly new releases that had significant major changes to the architecture.

1693
04:19:50.870 --> 04:20:13.310
senate chamber: But since then not a lot has happened on that front. Right? I still teach with the 2,008 book, and that's pretty good for most of the stuff in Act R. To tell them to put stuff in the imaginal module instead of all in the gold module. But largely, it's not changed that much from the theory end of it. But the software has changed a lot

1694
04:20:13.390 --> 04:20:33.940
senate chamber: right? And I think that from my perspective, the biggest change in the software is that Dan has. It's not a verb, but he has Api eyes, Dick right? And you can build a module or a device, or something in a language that isn't lisp. And you don't even have to know how actor is implemented to do it.

1695
04:20:34.030 --> 04:20:52.540
senate chamber: and most of the stuff that I've seen that's out there is in Python, which is sensible, since Python is now the most common programming language. But that's a really important step that I think we should keep in mind going forward. I think the model going forward for Act R should be. R,

1696
04:20:52.760 --> 04:21:15.309
senate chamber: all right. Interestingly, I think this one of the things that always comes up with future vector. Workshop is this question earlier of like, should it be at lisp anymore, or whatever anybody know the actual language in which R is implemented? C. It's a mixture of C and Fortran, all right. If R. Can still be using Fortran, we can still use it.

1697
04:21:15.310 --> 04:21:40.529
senate chamber: Now, the argument that I really want to make is it doesn't actually really matter what the core language is underneath the hood. A very small number of people need to touch that. I will say that if we think performance is really important, I'm going to recommend against Python because Python is a dog. It still is. It's really funny, because everybody used to hate list, because lisp was small. Well, lisp is actually now really fast, like list compilers are really good now.

1698
04:21:40.570 --> 04:21:43.580
senate chamber: and python is not fast.

1699
04:21:44.030 --> 04:21:53.790
senate chamber: The other thing about R that I like is that we have core team leadership. Currently, we are on essentially the benevolent dictators for life.

1700
04:21:54.020 --> 04:22:09.209
senate chamber: right? And that probably that's not sustainable forever going forward. And and this idea of sort of a core team that's responsible for the core functionality is a good way to go. But I think the thing that I really like about R is

1701
04:22:09.210 --> 04:22:31.959
senate chamber: that R is really a mixed model for software development. The base software is top down and the core team is responsible for that. But you can write stuff for R without being a member of the R. Core team that you can dump out and anybody can use it. Nobody knows what language. You wrote it in right. It doesn't matter. As long as it meets the R. Apis. It can be loaded into the system

1702
04:22:31.960 --> 04:22:43.570
senate chamber: right? I want another. I want a different vision module bam. It just loads the different vision module, and you don't care what the core system is written in or how it works right because it just works right

1703
04:22:43.570 --> 04:23:03.969
senate chamber: now. That's 1 of the things that I'm sure that the R. Team has worked on tremendously and is a lot of effort is their package management system is awesome because I don't know. I don't have any idea how it works. All I know is, it works for me every time I need it. I say, install module bam. I say library, bam, and it all just works right now.

1704
04:23:04.060 --> 04:23:11.949
senate chamber: if they can do it. Is there a reason we couldn't do it. And the foundation funding funding model also seems really good to me.

1705
04:23:12.110 --> 04:23:28.279
senate chamber: The other things that I'd like to see going forward. Out of this effort is tutorials, annotated examples for building new modules and devices would be really good. There's some of that sort of existing already, but I think it'd be cool if we could see that in more languages

1706
04:23:28.670 --> 04:23:33.389
senate chamber: also the same kinds of things for modifying internals. Right? So

1707
04:23:33.550 --> 04:23:55.739
senate chamber: it's hard to go to a meeting like this and not see somebody who's hacked the activation function, or some part of the Rl equation or some part of this icon. That's my personal favorite right? And so I think what we need is like an Api for hook functions so that you don't have to hack into act R to do this stuff right? You could just hit an Api, and you don't care what language is written in or how it works

1708
04:23:55.740 --> 04:24:06.990
senate chamber: right. The other thing I really want, because I'm an Hci guy is. I want actor to be able to talk to other things without me having to write a giant piece of software to make that happen every time I want to do it.

1709
04:24:06.990 --> 04:24:30.110
senate chamber: So the inability to connect act R to other systems is a big barrier. Right? Oh, well, you can just reimplement it in the actr. Environment is not the right answer, it may be for small psychology experiments. I want to interface with a flight simulator, right? Or a complicated piece of voting software, or whatever. I think that we need to figure out a way to support that. Better, to make that easier.

1710
04:24:30.110 --> 04:24:46.560
senate chamber: Maybe one of the ways we do it is we resurrect this thing. But Frank's not here that Rob's saying the one he worked on called Segman, which basically did machine vision right on a bitmap to get stuff in, like, you know, Opencd is pretty good. How hard.

1711
04:24:46.560 --> 04:25:06.439
senate chamber: how hard can it be, said Jeremy Clarkson. And one way we could support this is if we had a wrapper with same kind of software that I wrote for Mcl. For some kind of popular Ui package like Python, Tk. Or piglet, so that people had an example of. Here's how you do it. It might make it easier for people to build more of these things.

1712
04:25:06.900 --> 04:25:10.135
senate chamber: So there, that's like, really my 5 min on the nose.

1713
04:25:11.390 --> 04:25:17.049
senate chamber: Perfect. Do we have one question before we move on. We had a lot of good discussions in the chat.

1714
04:25:23.420 --> 04:25:28.349
senate chamber: Okay, in the interest of on time. True.

1715
04:25:32.150 --> 04:25:34.031
senate chamber: we're just gonna skip metal over here.

1716
04:25:35.240 --> 04:25:37.699
senate chamber: We're going to

1717
04:25:42.190 --> 04:25:53.489
senate chamber: all right, so I won't take up to 5 min, I'm sure, but I just want to say that I am honored to be a part of this pose effort to be a part of this project alongside.

1718
04:25:53.790 --> 04:26:03.610
senate chamber: like John and Kristen and Dan Morrison Andrea Kevin glove keep early and more tolerable.

1719
04:26:04.540 --> 04:26:10.050
senate chamber: Switch over here. And just say that you know

1720
04:26:11.190 --> 04:26:15.727
senate chamber: As part of this project, we are committed to learning and

1721
04:26:16.470 --> 04:26:25.139
senate chamber: about what the community wants out of actar. So you might probably see me over here typing a lot. I'm just trying to keep as much notes as possible.

1722
04:26:27.480 --> 04:26:31.550
senate chamber: and you know, we want to figure out, you know, what will make for

1723
04:26:32.020 --> 04:26:36.210
senate chamber: good governance, for having developed it.

1724
04:26:36.630 --> 04:26:41.020
senate chamber: Community, the user base things like that. So

1725
04:26:43.020 --> 04:26:49.240
senate chamber: looking forward to getting into this project and interviewing a lot of you know, all the involved is doing a lot.

1726
04:26:53.150 --> 04:26:56.469
senate chamber: I guess 2 things

1727
04:26:57.360 --> 04:27:00.979
senate chamber: that maybe we can get into a little bit of discussion.

1728
04:27:03.380 --> 04:27:10.760
senate chamber: Are is one kind of a vision, or maybe a desire

1729
04:27:11.170 --> 04:27:18.649
senate chamber: from a few of people on the team that we've mentioned as we're developing this project. And then another kind of

1730
04:27:19.200 --> 04:27:23.670
senate chamber: vision of mine of what I'd like to see in the future back there.

1731
04:27:24.536 --> 04:27:31.490
senate chamber: And it's what so my personal vision involves. A kind of

1732
04:27:32.520 --> 04:27:38.729
senate chamber: better, better way to handle model repository.

1733
04:27:39.110 --> 04:27:46.540
senate chamber: Right? We have the act on Web website, and it has a large list of related publications.

1734
04:27:46.860 --> 04:27:54.919
senate chamber: I think only a small subset of those actually have models attached to them, and many of those that actually do have models are

1735
04:27:55.170 --> 04:28:07.740
senate chamber: extremely old, Hector 4 even, or something models, and very hard to translate into, say, an actor seminal.

1736
04:28:08.010 --> 04:28:19.039
senate chamber: So but kind of so you know, updating that whole system of how we share models amongst each other, I think, could make a really really useful

1737
04:28:19.200 --> 04:28:32.140
senate chamber: effort in advancing science in general across. You know, we have all these tasks, all these models. But I've been doing a project recently where I've been trying to help source

1738
04:28:32.580 --> 04:28:36.309
senate chamber: existing models out there for various tasks, and

1739
04:28:36.430 --> 04:28:51.130
senate chamber: just finding them have been difficult. I can find some papers on some of these tasks that kind of describe these models sort of in some kind of way, but actually, implementations of them that I can just plug and play and get to work has been

1740
04:28:51.330 --> 04:28:55.459
senate chamber: extremely difficult to find. So you know, if we can come together

1741
04:28:55.640 --> 04:29:02.899
senate chamber: and say, Have a nice repository where we can all share our models willingly, easier, more easily.

1742
04:29:03.020 --> 04:29:03.960
senate chamber: Then.

1743
04:29:05.020 --> 04:29:26.490
senate chamber: you know, someone comes along with the model the same task. They can look at what's been done and maybe build off of it, you know. So if it was something like a Github repository or something. You can all contribute to those branches right and check out branches, modify things, test these test theories, build on theories and stuff. So I think that would be very useful.

1744
04:29:28.190 --> 04:29:31.790
senate chamber: So thing for the community

1745
04:29:31.930 --> 04:29:34.430
senate chamber: related to that would be

1746
04:29:35.600 --> 04:29:39.650
senate chamber: Similarly, you know, we've had in a history back door.

1747
04:29:39.840 --> 04:29:53.589
senate chamber: Dan has done a lot implementing actar and various modules and mechanisms, and but we've also had a lot of contributions from the community developing different modules and mechanisms.

1748
04:29:53.950 --> 04:30:02.920
senate chamber: Some of them get incorporated into the actor distribution. Some of the.

1749
04:30:04.160 --> 04:30:18.429
senate chamber: And then there's also been other kinds of kind of spawns of actor, and say, other languages are just kind of rebuilt or kind of, you know, parts of actor developed as other kinds of

1750
04:30:18.680 --> 04:30:19.660
senate chamber: architectures.

1751
04:30:21.380 --> 04:30:27.880
senate chamber: So you know again, there's some kind of github like repo, where you can

1752
04:30:29.850 --> 04:30:41.610
senate chamber: provide your contributions to actar and given enough consensus that could be merged into the main distribution. This would make things, I think, more accessible

1753
04:30:41.930 --> 04:30:48.350
senate chamber: and be able to build the science faster, much faster, that currently exists.

1754
04:30:49.285 --> 04:30:55.620
senate chamber: So that's kind of my personal visions of what I would like to see that I think could be useful for the community.

1755
04:30:55.800 --> 04:30:57.079
senate chamber: And then.

1756
04:30:57.744 --> 04:31:03.045
senate chamber: one thing that hasn't been mentioned yet. I'll go ahead and mention in case nobody else does, is

1757
04:31:05.370 --> 04:31:07.610
senate chamber: You know, you've talked about it a lot today is

1758
04:31:08.180 --> 04:31:13.040
senate chamber: figuring out how to ways to incorporate Gen. AI Llms with.

1759
04:31:13.210 --> 04:31:19.500
senate chamber: So I think that would be really

1760
04:31:20.540 --> 04:31:23.110
senate chamber: interesting. Path forward is to think about

1761
04:31:23.300 --> 04:31:42.959
senate chamber: what are ways we can reach out to these communities of users, expand our community base for people who would like to see that happen. And then also, how do we, you know, when we do have ideas, incorporate that and distribute that to the community, build on ways to leverage and Llms to attack R.

1762
04:31:43.580 --> 04:31:47.009
senate chamber: Build hybrid models, etcetera, that's all.

1763
04:31:50.660 --> 04:31:51.770
senate chamber: Thank you, Drew.

1764
04:31:51.880 --> 04:32:02.160
senate chamber: in the interest of keeping us on time. And because I think we're going to have a very spirited discussion at the end. Let's move right to Chris.

1765
04:32:04.930 --> 04:32:08.311
senate chamber: I hope that wasn't forecasting what I'm gonna talk about.

1766
04:32:09.300 --> 04:32:13.390
senate chamber: So I don't think this will be controversial, though maybe it will be.

1767
04:32:14.750 --> 04:32:26.679
senate chamber: So here goes. I'm really focused on the integration scale pipelining new domain capabilities.

1768
04:32:27.000 --> 04:32:39.200
senate chamber: And in the lab, we have this real need and growing need for the ability to conduct modeling, simulation and analysis across multiple levels of analysis.

1769
04:32:39.700 --> 04:32:51.350
senate chamber: So if you can imagine a pyramid for a second at its foundation is what we do in the actar community. We develop high fidelity models.

1770
04:32:51.450 --> 04:33:02.899
senate chamber: Sometimes they're referred to as physics, level models, very detailed, very accurate. And it's typically focused on the components of the system.

1771
04:33:03.490 --> 04:33:30.819
senate chamber: We move up a level, we get into kind of an engagement level where it's kind of a 1-on-one situation. This is what Mike Byrne brought into the architecture, right, the ability to interact with the external environment. Now you have this one-on-one engagement capability up another level is Mini on Mini. We refer to this as a mission level in the Air Force, and then above that, is the campaign level, hundreds or thousands

1772
04:33:31.110 --> 04:33:33.870
senate chamber: on each other right interacting with each other.

1773
04:33:34.570 --> 04:33:35.740
senate chamber: How do we do that?

1774
04:33:36.570 --> 04:33:37.290
senate chamber: Right?

1775
04:33:37.619 --> 04:33:47.139
senate chamber: So what I'm about to say might be a suggestion for a direction or a capability pulled into pose.

1776
04:33:47.540 --> 04:33:55.379
senate chamber: And I'll use our problem as a microcosm of what actor might deal with in the future.

1777
04:33:56.000 --> 04:34:06.389
senate chamber: So we have this need for these analyses, and our leadership is making decisions about what to invest in as a function of the results of these analyses.

1778
04:34:06.660 --> 04:34:10.889
senate chamber: Guess what is missing. In nearly all of these analyses.

1779
04:34:11.320 --> 04:34:16.143
senate chamber: Anybody want to take a guess, not Alex.

1780
04:34:17.210 --> 04:34:35.660
senate chamber: Pardon people. Thank you. Human representation is devoid, and we know that it impacts the effects of these systems as we scale. And so we've been working to get systems, human representation into these types of analyses to inform

1781
04:34:36.840 --> 04:34:42.989
senate chamber: strategy both in terms of R&D, and and the distribution of funding.

1782
04:34:43.230 --> 04:34:53.780
senate chamber: How are we doing that? We have pulled models from across the directorate from physiology, and my directorate is the human effectiveness directorate

1783
04:34:53.980 --> 04:35:11.370
senate chamber: in the Air Force Research Laboratory. So across physiology, perception and cognition. So there's modeling going on all over this space. But as Drew was mentioning just a second ago. There's not a repository for these models.

1784
04:35:11.710 --> 04:35:18.780
senate chamber: so we need some way to pull this information together. That is then of use to my leadership

1785
04:35:19.200 --> 04:35:23.770
senate chamber: a la a repository or an architecture.

1786
04:35:24.180 --> 04:35:29.670
senate chamber: So we've pulled together a representation of human physiology.

1787
04:35:29.840 --> 04:35:39.209
senate chamber: perception and cognition based on the common common model of cognition developed within a model-based systems engineering framework.

1788
04:35:39.710 --> 04:35:46.160
senate chamber: So this is simply a framework that provides information about relationships and constraints

1789
04:35:46.490 --> 04:35:57.080
senate chamber: at different levels of the system, and is often used as it for design, visualization, documentation, and specification of complex systems.

1790
04:35:58.119 --> 04:36:11.270
senate chamber: So why did we do this? Well, first, st we don't want to be in the business of writing code for all of the systems. Analysis, what we want to be able to do is say, here is a human representation.

1791
04:36:11.470 --> 04:36:22.629
senate chamber: It's specified as such a way, and we will help you get that code implemented into your modeling and simulation formalism which allows us to be language agnostic.

1792
04:36:23.500 --> 04:36:34.870
senate chamber: right and and from there. We've had some successes. So we are getting into systems analysis at the

1793
04:36:35.880 --> 04:36:45.140
senate chamber: mission level and at the campaign level. But the challenge here is scaling. So typically macro models are built

1794
04:36:45.520 --> 04:36:47.040
senate chamber: of a task.

1795
04:36:47.250 --> 04:36:58.049
senate chamber: This is the thing Act R is going to do. And these are the predictions it provides with which we compare against human cognition. And then we have a fit. Then that adds back to the theory.

1796
04:36:59.210 --> 04:37:06.289
senate chamber: What we need as we scale is the ability to represent the constraints those theories have on information processing

1797
04:37:06.419 --> 04:37:12.360
senate chamber: as opposed to developing a model of a particular task. So going back up to the campaign level.

1798
04:37:12.779 --> 04:37:25.279
senate chamber: We don't want to build a thousand different models for a thousand different tasks that then have to interact with each other. I mean, it's just not going to work. But we do want to be able to make

1799
04:37:25.450 --> 04:37:33.080
senate chamber: clear recommendations about changes as a function of failures that you might observe in these types of tasks.

1800
04:37:33.390 --> 04:37:40.100
senate chamber: the rate with which information is processed by individuals at these levels of of the task, so that you can have

1801
04:37:40.250 --> 04:37:53.109
senate chamber: accurate results as a function of the analysis, and they can make the correct or good decision on which R. And D should be focused on, and which funding should be provided.

1802
04:37:53.710 --> 04:38:02.469
senate chamber: And I just one last thing that image you saw. I am much younger than the picture.

1803
04:38:03.470 --> 04:38:04.189
senate chamber: Thanks.

1804
04:38:05.830 --> 04:38:09.640
senate chamber: Thank you, Chris. Natalie, let's keep it rolling.

1805
04:38:12.230 --> 04:38:25.439
senate chamber: Okay, I also provided some slides, because, okay.

1806
04:38:30.330 --> 04:38:31.160
senate chamber: okay.

1807
04:38:34.720 --> 04:38:46.190
senate chamber: yeah. So my thoughts go more into the direction. 1st of all, where do we want to go? Do we want to get other people involved. And what would you really need for that?

1808
04:38:46.450 --> 04:38:50.909
senate chamber: So what other research areas would benefit from?

1809
04:38:51.410 --> 04:38:58.980
senate chamber: And I think that this interactive learning project. And the book that was

1810
04:38:59.220 --> 04:39:23.679
senate chamber: yeah, part of what happened. There is something quite relevant, and it's very relevant to a lot of different disciplines. And you might want to reach out or provide something that would be helpful for them to use. Maybe some simplified platform that would be easier to add to simulations to systems, to agent environments.

1811
04:39:23.910 --> 04:39:27.989
senate chamber: And maybe that would be an interesting way to go.

1812
04:39:28.790 --> 04:39:36.329
senate chamber: So and I think that is a relevant challenge, not just for

1813
04:39:36.619 --> 04:39:50.009
senate chamber: robotics and agent intelligent agents, but also for psychology. We've heard about fear of mind teaming, cooperation. All these aspects could benefit a lot from

1814
04:39:50.210 --> 04:39:51.310
senate chamber: simple

1815
04:39:51.680 --> 04:39:59.769
senate chamber: form of an AR platform that could be easily used. And you do not need to go too much into detail.

1816
04:39:59.950 --> 04:40:05.067
senate chamber: But actually, so I 1st want to cover actually watch.

1817
04:40:05.600 --> 04:40:21.460
senate chamber: we just heard from Chris and the different levels we are usually working on with the architecture. So some of us work on specific aspects of the architecture on modules and buffers. Some go even deeper and look at specific parameters.

1818
04:40:21.580 --> 04:40:26.720
senate chamber: So actually, a lot of the specific elements

1819
04:40:26.960 --> 04:40:46.979
senate chamber: that was just named, and others are more interested in how the different modules work together. And then, again, we have the comparison with the human brain. And what else do I have. Oh, that is also something, I think, that could be relevant for a lot of other disciplines.

1820
04:40:47.160 --> 04:41:07.520
senate chamber: So it's about metacognition. For example, we've worked a lot on the sense of control in interaction, and we added a semi-modal level to act on the lower and higher cognitive level and sense of control that we needed to cope with very dynamic and unforeseeable environments.

1821
04:41:07.650 --> 04:41:25.259
senate chamber: sense of self, kind of. And I think I've heard so many talks about that topic, and that would also be something that could be taken or could be investigated from other disciplines on a more simpler level.

1822
04:41:25.650 --> 04:41:29.730
senate chamber: If we have a such a platform

1823
04:41:30.290 --> 04:41:37.620
senate chamber: easy to use and to connect, and maybe also easier to to provide support.

1824
04:41:38.850 --> 04:41:40.372
senate chamber: So actually, that's

1825
04:41:41.120 --> 04:41:59.359
senate chamber: picture of an interactive task running agent. So maybe we've talked also about perceptual modules. So if we use intelligent agents in environments, or if we use some kind of robotic system, we do not need, or we do not

1826
04:41:59.360 --> 04:42:23.530
senate chamber: be so detailed about the perceptual modules we could get information in, but still concentrate on representations that are built up that are updated or that might cause problems. So those problems that bring us to have some false beliefs and stuff like that situation awareness, maybe also build personal models in interaction.

1827
04:42:23.660 --> 04:42:25.364
senate chamber: And yeah.

1828
04:42:26.150 --> 04:42:35.119
senate chamber: so I think I really like that, and I think we should definitely look rather into the direction of AI, and also what

1829
04:42:35.230 --> 04:42:42.379
senate chamber: current AI approaches are not good at. And maybe we can bring.

1830
04:42:43.370 --> 04:42:46.425
senate chamber: And AI psych psychological

1831
04:42:47.400 --> 04:42:56.760
senate chamber: scientists and are closer together. And yeah, find a solution to such a thing, because there are so many

1832
04:42:56.970 --> 04:43:07.589
senate chamber: very interesting areas to bring more attention to the theory and still have systems that are easier to

1833
04:43:07.770 --> 04:43:16.560
senate chamber: taken care of for those that are not experts of AR. So that would maybe mean less work.

1834
04:43:16.760 --> 04:43:18.740
senate chamber: because most of the work will probably

1835
04:43:18.880 --> 04:43:24.419
senate chamber: from students and from people that really don't use the architecture. But I'm not sure about that.

1836
04:43:25.200 --> 04:43:26.155
senate chamber: So

1837
04:43:28.960 --> 04:43:48.880
senate chamber: so what are relevant areas like? I said that. So I think it would be good or a thing to think about, to have a simplified approach that others could use more easily. For example, for students, they really have big problems to get to wrap their head around

1838
04:43:49.000 --> 04:44:03.940
senate chamber: all the aspects of cognitive modeling. So it's much easier. And that's what I'm doing to get them involved in providing some agent environments and tasks, and they don't need to have the whole it teacher, to learn all the aspects.

1839
04:44:04.070 --> 04:44:07.790
senate chamber: and still could understand a lot what cognitive

1840
04:44:08.500 --> 04:44:14.650
senate chamber: science on human cognition really provides and helps for such.

1841
04:44:17.310 --> 04:44:29.629
senate chamber: okay. And I think also to go to these areas would give us more, maybe more foundation to get more funding and also to have other areas

1842
04:44:29.750 --> 04:44:37.320
senate chamber: that would also help in to to spot such a self contained structure.

1843
04:44:39.060 --> 04:44:47.569
senate chamber: Okay, yeah, okay, that we already talked about all these drawbacks of current AI systems.

1844
04:44:49.980 --> 04:44:58.419
senate chamber: Yeah, so that is just one example. That's 1 example that I'm working on a lot is to.

1845
04:44:58.540 --> 04:45:05.559
senate chamber: I like to bring architecture into the mind of embodied agents.

1846
04:45:05.740 --> 04:45:13.020
senate chamber: and we've done that on a simple version, and also added large language model for the interaction for the dialogues, but not for the

1847
04:45:13.280 --> 04:45:14.140
senate chamber: presentations.

1848
04:45:14.240 --> 04:45:22.999
senate chamber: And that is enabled on us for to work on very interesting problems like operations. And yeah.

1849
04:45:25.560 --> 04:45:26.700
senate chamber: yeah, thank you.

1850
04:45:28.520 --> 04:45:29.710
senate chamber: Thanks so much, Natalie.

1851
04:45:30.020 --> 04:45:33.860
senate chamber: Let's keep. Let's keep moving on, Dario.

1852
04:45:37.160 --> 04:45:44.930
Dario Salvucci: So I'll try to be brief. And today and others have already covered so many great points. You know, to me the summary is really

1853
04:45:45.050 --> 04:46:02.189
Dario Salvucci: doing work like this is a lot of work. Honestly, it's not always appreciated. So it's really valuable. For all the reasons that everyone has mentioned from community to practical reasons, etc. I really just want to make one larger point. So

1854
04:46:02.490 --> 04:46:26.050
Dario Salvucci: in the Venn diagram actor wasn't in the language bubble, and it's definitely not a language like Python or R. But I would say that it has at least one foot inside that bubble. Well, Act R. Doesn't have feet, but if it had a foot it would have one foot in there, and and to me it says something about the relationship to programming languages. Mike touched on this already a little bit.

1855
04:46:26.050 --> 04:46:39.760
Dario Salvucci: The actar compiler has already been implemented in many languages, and in the end, if the compiler is doing its job, it doesn't matter what language the compiler's written in it runs the actar model as it should.

1856
04:46:39.760 --> 04:46:52.630
Dario Salvucci: But at least for me personally, it's always the reason for choosing a particular language is not really act R directly, but the task simulation, environment. And I know that

1857
04:46:52.640 --> 04:47:06.809
Dario Salvucci: we don't need to write the test simulation environment in the same language. We can hook it up via network sockets whatever, but often it's easiest to do it in the same language, and just especially from a tutorial standpoint to have things running.

1858
04:47:07.010 --> 04:47:27.719
Dario Salvucci: So I like how Christian presented those 4 types of contributions of architecture, modules, models, and applications, and I think I would break down the applications, at least into a couple subcategories where you do have application domains that are general areas to apply these models, but also

1859
04:47:27.720 --> 04:47:57.630
Dario Salvucci: very specifically, the tasks and the code that runs particular tasks is super important for the kinds of things that we do. And also, I think, really important for people to be able to pick up models and get them running quickly in the same way that data science folks can do a tutorial on neural networks and write a few lines of code and download not only the architecture, so to speak, but also the data sets and run them immediately and get something.

1860
04:47:57.770 --> 04:48:13.739
Dario Salvucci: just as a quick example, the most popular programming language might be python. But if we think about user interfaces that are built by far the most popular user interface right now is websites. HTML, css, javascript. So

1861
04:48:14.220 --> 04:48:39.090
Dario Salvucci: if we offer people models that interact with the kinds of things that they're already used to building, or they already have built like websites, then that makes it that much easier for them to interact with it. And I know this idea of actor models interacting with web pages is at least 2 decades old, if not older. But it's something that the more we make it easier the more people can can get into this.

1862
04:48:39.416 --> 04:48:47.253
Dario Salvucci: So I think the bottom line is really to to think about these task environments as in a way like 1st class citizens of

1863
04:48:47.810 --> 04:48:56.209
Dario Salvucci: any act, our ecosystem, along with the usual architecture and and models, and everything else that that folks have said. So thank you.

1864
04:48:59.060 --> 04:49:00.410
senate chamber: Great. Thank you so much.

1865
04:49:00.550 --> 04:49:02.899
senate chamber: Last speaker, Henrik.

1866
04:49:05.088 --> 04:49:09.549
Hedderik van Rijn: Yeah, thank you so much. And great to hear all these different ideas.

1867
04:49:09.660 --> 04:49:10.350
Hedderik van Rijn: So

1868
04:49:11.210 --> 04:49:31.309
Hedderik van Rijn: even though I could also approach this from the more academic stand I've been trying to really sort of like. Think about what my contributions would be from a more entrepreneurial stance, and one of the things that I find very interesting is that we are sort of like that. I hear all these references to to large language models.

1869
04:49:31.820 --> 04:49:51.909
Hedderik van Rijn: Just the last 2 weeks I was at AI Educational data mining, and we were presenting at our base work, and people truly were thrilled that we were presenting something. That's not an Llm. And it was multiple reasons. And I think that one of the reasons was that we actually know what our models are doing.

1870
04:49:52.360 --> 04:50:07.010
Hedderik van Rijn: And this is, of course, so obviously, I'm sort of like preaching to the choir here. But there's a very, very, very important component in this that will be extremely important. For from for any small startup that wants to

1871
04:50:07.586 --> 04:50:13.969
Hedderik van Rijn: get a foothold in Europe because we get the the European Union, AI Act.

1872
04:50:14.170 --> 04:50:22.389
Hedderik van Rijn: any system that makes a decision that has an impact on a life of an EU citizen needs to be explainable.

1873
04:50:22.820 --> 04:50:26.579
Hedderik van Rijn: And that's impossible almost for these Llms.

1874
04:50:26.630 --> 04:50:54.249
Hedderik van Rijn: And obviously, if you have enough lawyers, and you have enough sort of like funding to get that through. You probably will get that through. But we, as a small company, actually managed to get AI act approvals because I can show what my model does. I can tell it what the input parameters are and how it combines. And there might be some noise in there. But good lawyers are okay with noise. They know that this is how a system works, so I will never move to an Llm.

1875
04:50:54.310 --> 04:50:59.289
Hedderik van Rijn: Because that will get me into enormous trouble with the with the AI Act.

1876
04:51:00.130 --> 04:51:10.170
Hedderik van Rijn: Now, a completely different perspective is that I also heard many people talk about. We need to allow others to interface into act R,

1877
04:51:10.870 --> 04:51:16.249
Hedderik van Rijn: but that means that we describe act R as the sort of like as the code base

1878
04:51:16.730 --> 04:51:38.669
Hedderik van Rijn: for me act R is much more like the theory, the idea, the concepts that we all share the ideas of retrieving information from memory, of of getting information in and out of the buffers, and that can be implemented, of course, in the great system that Dan and others have been building. But it's also much more of a general idea.

1879
04:51:39.110 --> 04:51:49.550
Hedderik van Rijn: And I don't want my programmers to need to worry about any potential update in a particular module that might mess with my system.

1880
04:51:50.070 --> 04:51:55.720
Hedderik van Rijn: So instead of going to a more sort of like huge cognitive system.

1881
04:51:55.950 --> 04:52:00.810
Hedderik van Rijn: I would say that we need to think about the atomic components of that system.

1882
04:52:01.630 --> 04:52:14.799
Hedderik van Rijn: The declarative memory system is the thing that we use, and I would love to be able to sort of like keep using the the most recent update, and actually have my my programmers contribute to that module.

1883
04:52:15.240 --> 04:52:25.029
Hedderik van Rijn: But I don't want them to need to worry about what might change if other components also change. So I would actually argue that from an from an

1884
04:52:25.210 --> 04:52:27.530
Hedderik van Rijn: small company's perspective.

1885
04:52:27.800 --> 04:52:36.059
Hedderik van Rijn: a more integrative whole, and even sort of like communicating with hooks and that kind of stuff that's just too difficult. It should be simple and small.

1886
04:52:36.630 --> 04:53:05.559
Hedderik van Rijn: We right now have code in typescript, in Javascript, in Python, or sort of like certain declarative memory systems. But at the moment that is sort of like void with published it, and it might be somewhere on Osf or on one of the other systems, but I think that there really would be an benefit to a more modular system, much more modular even than that we're talking about now with different modules that can be added

1887
04:53:06.440 --> 04:53:10.360
Hedderik van Rijn: now. And as the very final point about financing

1888
04:53:11.028 --> 04:53:26.799
Hedderik van Rijn: I think that many of these other organizations have moved to some sort of like support tickets. And the reason for that is that well, it's not that the companies that I'm involved in are doing that. Well, but even if they were, we couldn't just give money.

1889
04:53:27.000 --> 04:53:34.119
Hedderik van Rijn: My investors would never agree with us just giving money to something that is also available just online.

1890
04:53:34.550 --> 04:53:54.019
Hedderik van Rijn: So for that there definitely should be some sort of support system or support tickets, or whatever else is needed that I would be able to list in my commercial grants, or that I could explain to my to my investors that were actually buying something for that money.

1891
04:53:54.990 --> 04:54:10.319
Hedderik van Rijn: and I think that these were sort of like my main ideas. I think it's great that we're not doing much with Llms that we really need to think about what the separate components are that companies might be interested in using because they don't care about the full system.

1892
04:54:10.480 --> 04:54:20.520
Hedderik van Rijn: And what is the best way to actually get some financing in that in the end might up to something that could potentially fund the whole system. Thank you.

1893
04:54:23.710 --> 04:54:28.570
senate chamber: Thank you, Henrik. Did we get a round for all the speakers so far?

1894
04:54:34.990 --> 04:54:44.270
senate chamber: Yeah. So who's going to create the Bitcoin Dan coin that brings them to the top of the patch list

1895
04:54:45.070 --> 04:54:46.170
senate chamber: tech support.

1896
04:54:48.880 --> 04:54:53.179
senate chamber: Okay, do we have any questions for the group at large?

1897
04:54:54.390 --> 04:54:55.170
senate chamber: Comment.

1898
04:54:55.930 --> 04:55:08.729
senate chamber: So 1st of all, from a organization point of view, I think. Let's not worry about rushing. We can just keep going as long as we want the the schedule. There is arbitrary. And yeah, we're just gonna keep going. So

1899
04:55:09.090 --> 04:55:10.340
senate chamber: not as long.

1900
04:55:10.720 --> 04:55:19.740
senate chamber: 1 point that multiple speaker mentioned and Henrik just mentioned it at the end about his more modular system is that

1901
04:55:21.050 --> 04:55:24.499
senate chamber: contributions right? Well, model? Certainly.

1902
04:55:24.870 --> 04:55:25.375
senate chamber: Oh.

1903
04:55:26.360 --> 04:55:45.140
senate chamber: disposable, not composable. You create a model, and it's fantastic, and you spend months or years of your life on it, and it's great can account for the data, but it's almost never the case that somebody can reuse it and compose it with another model to do something more complex. And that's the hallmark of software.

1904
04:55:45.280 --> 04:55:53.509
senate chamber: right? The ability to create stack libraries on top of libraries. And that's a methodological issue that we have not

1905
04:55:53.750 --> 04:56:05.790
senate chamber: solve that part. I think that's going to be a fundamental issue, that we're going to need to address one way or another to sort of grow the ecosystem. It's got to be reusable. It's got to be composable

1906
04:56:05.950 --> 04:56:09.050
senate chamber: again. That's something to work on.

1907
04:56:15.490 --> 04:56:26.160
senate chamber: Yeah, in the firm, like Henric's suggestion or- or idea about the transparency.

1908
04:56:26.310 --> 04:56:32.639
senate chamber: So here we're training up transparency

1909
04:56:33.040 --> 04:56:39.569
senate chamber: of a bespoke system which is a car for the statistical power

1910
04:56:40.100 --> 04:56:44.060
senate chamber: advantages of a brute force system, which is an Llm.

1911
04:56:46.140 --> 04:56:53.869
senate chamber: But I believe we can do. We can make use of both, and not lose

1912
04:56:54.090 --> 04:56:57.490
senate chamber: the advantages of transparency, and having a scope system.

1913
04:56:58.140 --> 04:57:04.530
senate chamber: We can, for example, one of the examples that were given here were, to use an Llm.

1914
04:57:04.730 --> 04:57:06.879
senate chamber: Populates the detractive memory.

1915
04:57:07.700 --> 04:57:15.690
senate chamber: The system still is transparent, but it uses the the force going on

1916
04:57:21.990 --> 04:57:23.190
senate chamber: any comments.

1917
04:57:24.950 --> 04:57:37.149
senate chamber: Arnas, I found a stay from Penn State. Thanks for bringing that up as Xr models need to interact

1918
04:57:37.340 --> 04:57:44.479
senate chamber: with user interfaces and the same task environment as participants in the study interact with.

1919
04:57:44.670 --> 04:57:54.000
senate chamber: During my Phd, I worked on eyes and hands model that are cognitive models that can interact with the task environment.

1920
04:57:54.120 --> 04:58:15.580
senate chamber: I developed a tool. I call it J. Segment inspired by segment, Java segmentation and manipulation. I presented at Iccm at a workshop. And currently a graduate student in my lab is also working on a similar tool. We call it visitor vision plus model.

1921
04:58:16.150 --> 04:58:26.060
senate chamber: So my question is, is there a path for these tools to become as part of actual package, and how we can

1922
04:58:26.180 --> 04:58:31.739
senate chamber: like promote them? So other researchers know about them and use them.

1923
04:58:32.240 --> 04:58:37.049
senate chamber: During my Phd. I found many, many works in this line.

1924
04:58:37.290 --> 04:58:53.700
senate chamber: I think they were hidden kind of. They started as just a paper, and no one continue on that work. We can repurpose these works and how we can promote them, how we can add them

1925
04:58:53.920 --> 04:58:55.910
senate chamber: to the isol pack.

1926
04:59:06.810 --> 04:59:15.633
senate chamber: Yeah, I think that would be a really good opportunity for us is like I mentioned earlier, right?

1927
04:59:17.680 --> 04:59:21.710
senate chamber: definitely need to move the I think

1928
04:59:21.890 --> 04:59:27.509
senate chamber: I think I don't know if anything's this. But we need to move the act are

1929
04:59:27.960 --> 04:59:37.909
senate chamber: distribution into the 21st century, using things that other software uses like github, or other gateway

1930
04:59:38.780 --> 04:59:42.540
senate chamber: where you can have a version history.

1931
04:59:43.310 --> 04:59:46.950
senate chamber: You can have branches. You can have

1932
04:59:47.070 --> 04:59:53.970
senate chamber: additional modules or mechanisms right? That you could load in there all in kind of a

1933
04:59:54.180 --> 04:59:58.760
senate chamber: one. Stop shop, if you will, that anybody could go to search for

1934
04:59:58.900 --> 05:00:02.759
senate chamber: what they're interested or or working on, and find related

1935
05:00:02.990 --> 05:00:05.589
senate chamber: things that people other people have been working on.

1936
05:00:05.800 --> 05:00:09.730
senate chamber: And then you can expand upon that incrementally.

1937
05:00:09.970 --> 05:00:11.114
senate chamber: That's how it should.

1938
05:00:12.040 --> 05:00:15.540
senate chamber: Right? Definitely, this is a great idea people.

1939
05:00:16.820 --> 05:00:19.110
senate chamber: So I'll add my 2 cents.

1940
05:00:20.250 --> 05:00:25.710
senate chamber: There is a process for getting these systems or the subsystems pulled into the architecture.

1941
05:00:25.940 --> 05:00:29.499
senate chamber: I suspect not. Everybody knows what that process is.

1942
05:00:29.730 --> 05:00:32.330
senate chamber: So moving to an open source approach

1943
05:00:32.560 --> 05:00:35.350
senate chamber: would push the community to dictate

1944
05:00:35.520 --> 05:00:44.030
senate chamber: and be clear about what that process is to everybody who wants to contribute. So moving to an open source system will help push us in that direction.

1945
05:00:44.320 --> 05:01:00.030
senate chamber: Promoting it is a completely different story. I mean, that's kind of on the author. Right? So the person who developed the system needs to get out in front of us, and lots of other people to talk about how excellent their new system is, and why everybody should be using it.

1946
05:01:00.210 --> 05:01:10.359
senate chamber: That's that's on the author. And then I suspect all of the hidden ones you were talking about to address. That is that some of these things just

1947
05:01:10.660 --> 05:01:24.180
senate chamber: unfortunately die on the vine. It's a good idea they've taken it so far. They've lost interest in that particular direction of research. There's no funding for that direction of research anymore. And that's the way it goes. Sometimes

1948
05:01:28.150 --> 05:01:57.209
senate chamber: I mean, I think that the our package model also supports that kind of idea. Right? I mean, if you have it right, you can always go. Look at what I mean. The list is on cram. Right? Here's all the things I can go install package, and I get it right, and I occasionally find things on there that I thought I was going to have to write, you know, so I think that kind of model, whether it's Github or whether it's the arson I mean. All these kinds of models, where you have

1949
05:01:57.260 --> 05:02:00.990
senate chamber: the ability to load additional things that are

1950
05:02:01.250 --> 05:02:07.550
senate chamber: at least a little bit curated. Right would would serve the whole community. Well.

1951
05:02:09.910 --> 05:02:18.140
senate chamber: yes, I guess that is one of the points of the the pose process. Right is building up. That that's what the the initial pose Grant is for is to

1952
05:02:18.550 --> 05:02:20.300
senate chamber: flush out what the.

1953
05:02:20.480 --> 05:02:28.959
senate chamber: what the system we should have for that is. So I agree there should be other systems. I don't know what they should be. Exactly

1954
05:02:30.230 --> 05:02:45.719
senate chamber: perhaps the reason we're not on Github is just momentum. We've been in subversion since before git existed. So not Github, I mean literally git as a repository system. So we have branches and histories that

1955
05:02:45.910 --> 05:02:48.899
senate chamber: go back to 2,000 and

1956
05:02:49.060 --> 05:02:58.610
senate chamber: 5, I believe, for actor, you know. 6. So it's just it's never moved over because we have a system that's been built up. And

1957
05:02:59.100 --> 05:03:04.230
senate chamber: at some point it should. I understand that. But it hasn't.

1958
05:03:05.230 --> 05:03:09.689
senate chamber: And what was the management system that Christian and I and you used for Act R. 5.

1959
05:03:16.960 --> 05:03:18.480
senate chamber: I have a question for Chris.

1960
05:03:19.510 --> 05:03:22.489
senate chamber: So the leadership and funding decisions. How

1961
05:03:23.020 --> 05:03:33.089
senate chamber: how often or how much do they weigh validation against experimental data or other kinds of real world data in general within that process? And I think of this within the Bose context, because

1962
05:03:33.220 --> 05:03:37.520
senate chamber: expanding, depending on what the expansions for or the

1963
05:03:37.770 --> 05:03:40.810
senate chamber: community is for it's for potential people who may.

1964
05:03:41.100 --> 05:03:42.850
senate chamber: You know large funding users we send.

1965
05:03:43.420 --> 05:03:50.160
senate chamber: They might ask that question, should that be rolled into how we expand that that makes sense.

1966
05:03:50.520 --> 05:03:53.079
senate chamber: I think there were a couple in there.

1967
05:03:55.050 --> 05:03:56.509
senate chamber: and by that I mean, I don't know.

1968
05:03:56.640 --> 05:03:59.280
senate chamber: But yeah, I'll try to address it. So

1969
05:04:00.330 --> 05:04:11.519
senate chamber: so I think the 1st question was, how important is validation of the underlying models that these analyses use for making budgetary and R&D decisions.

1970
05:04:11.810 --> 05:04:22.280
senate chamber: And it really depends on the person who is being presented the analysis. I've seen some instances where the General has drilled all the way down into

1971
05:04:22.500 --> 05:04:27.700
senate chamber: very explicit details of the models that underlie results on a Pareto front.

1972
05:04:28.170 --> 05:04:30.519
senate chamber: and I've seen people just take it

1973
05:04:31.130 --> 05:04:50.760
senate chamber: as verbatim from the person presenting the information. So it really depends. The group I've been working with most recently has a mantra of really validate and verify at the physics level. So we can be confident at higher levels of analysis.

1974
05:04:51.050 --> 05:05:09.289
senate chamber: And my my poke to them. Is that just because it's worked at this lower level of analysis doesn't necessarily mean it's valid at scale, and work needs to be done there as well, and so I think I'm slowly getting that idea into mind. But I don't know if that's a

1975
05:05:09.640 --> 05:05:12.789
senate chamber: an answer you wanted to hear. But I think that's kind of where things stand.

1976
05:05:13.170 --> 05:05:21.299
senate chamber: Yeah, no, that that it answers the question. But I was also putting it for the post community is that part of the ecosystem developing

1977
05:05:21.840 --> 05:05:25.519
senate chamber: because some people, if you're trying to expand it, one of the 1st questions will be.

1978
05:05:25.760 --> 05:05:28.460
senate chamber: why do we care about that?

1979
05:05:28.750 --> 05:05:31.430
senate chamber: Do this job? And you're not going to believe you unless you have value.

1980
05:05:31.780 --> 05:05:32.479
senate chamber: It's some.

1981
05:05:35.250 --> 05:05:42.970
senate chamber: So maybe you're suggesting, along with a code in theory, repository in subversion.

1982
05:05:44.520 --> 05:05:53.350
senate chamber: a paper repository and a data repository that provide evidence for the support for the code and the theory.

1983
05:05:53.900 --> 05:05:57.749
senate chamber: Something like that. Yeah, I I think that's a very good idea.

1984
05:06:03.590 --> 05:06:26.129
senate chamber: I think that's a very relevant point to make to also be able to publish environment simulations. Maybe a simple task model, because that would help people to get started, and also to reference what model you're working on. And I think that actually should also be part of.

1985
05:06:26.340 --> 05:06:32.089
senate chamber: So you can publish your simulation with a model and

1986
05:06:32.290 --> 05:06:41.420
senate chamber: have other people use it, and maybe and so that would also help people to get started model and

1987
05:06:41.750 --> 05:06:51.480
senate chamber: not trying to build everything new. But if you already have a very complex model, nobody will really understand it so. But that could be a special form. What

1988
05:06:51.630 --> 05:06:55.350
senate chamber: be necessary to do such a vacation.

1989
05:06:55.620 --> 05:06:58.120
senate chamber: and I think that would really

1990
05:06:58.250 --> 05:07:08.080
senate chamber: help us in a lot of more applied areas where we attach acts to measure white environments.

1991
05:07:08.540 --> 05:07:14.299
senate chamber: Give us a good take on that, I dare you.

1992
05:07:15.610 --> 05:07:39.019
Dario Salvucci: Yeah, I just wanted to throw out there that I think one of the big questions, the big challenges with a project like this is to really figure out who the audience is. And I think there are people. If the audience is people developing cognitive theories to publish academic papers, that's an audience that I think we're all familiar with, and that audience

1993
05:07:39.020 --> 05:07:58.499
Dario Salvucci: cares about transparency. They care about understanding the models. They want faithfulness to empirical data. And so that leads you in a certain direction for this kind of project, I think, where the empirical data tied in with the models, for example, is very important.

1994
05:07:58.500 --> 05:08:21.930
Dario Salvucci: If you can easily imagine another audience that thinks of these models more like AI agents, more like Llms. They have a problem to solve. They want an agent to do some task, and they don't necessarily care that it does it in a human-like way, and they might not care as much about transparency or empirical backing.

1995
05:08:22.280 --> 05:08:41.019
Dario Salvucci: And I think, or at least in my mind, it's an open question of whether you can kind of serve both of those audiences with the same, with the same approach. I hope you can, because I think they're both great audiences, but I think the AI Agent audience is probably the bigger audience here.

1996
05:08:41.020 --> 05:08:59.300
Dario Salvucci: but it's possible that in in serving one it sort of weakens for the other, and I think that would be in terms of thinking about this project and the kind of exploration that's going to happen soon. That seems like a fundamental question of how to really focus something like this.

1997
05:09:03.850 --> 05:09:04.620
senate chamber: Andrea.

1998
05:09:07.959 --> 05:09:14.250
Andrea Stocco: I was just 1st of all, thanks for everybody's comments. I like literally taken like

1999
05:09:14.540 --> 05:09:20.119
Andrea Stocco: 6 or 7 pages of unwritten note, which I hope I'm able to decipher. At the end of this session

2000
05:09:20.570 --> 05:09:45.099
Andrea Stocco: I was really liking Daya's point, one of the things that we realize. We have a potential problem, not a potential problem, but a blind spot is, who are the users of Aktar. So we see the same people. We've seen the same people sometimes for decades, and it's a tight, knit community. Sometimes new people come in.

2001
05:09:45.380 --> 05:09:50.310
Andrea Stocco: but it's very clear from the data that them provide us, that there is a world out there

2002
05:09:50.700 --> 05:09:56.140
Andrea Stocco: of people who use Aktar that we didn't have an idea. I discovered people use Aktar at my own university.

2003
05:09:57.100 --> 05:09:58.550
Andrea Stocco: and

2004
05:09:58.730 --> 05:10:07.429
Andrea Stocco: and and they are squarely in this gigantic AI model. They just were looking for something. They found something. And they're like, Oh, yeah, this does exactly what they needed to do.

2005
05:10:07.950 --> 05:10:19.000
Andrea Stocco: and and clearly a bigger, a big point of our 1st steps. The biggest step, probably, is to have an idea of how many people are using a time for what.

2006
05:10:22.730 --> 05:10:24.520
senate chamber: Christian. Did you still want to talk?

2007
05:10:25.020 --> 05:10:31.710
senate chamber: Sure. So the the issue that Dario raised about different

2008
05:10:31.830 --> 05:10:40.019
senate chamber: different form factors, different requirements. I don't think it's unsolvable. So, for example, we've used Apta to

2009
05:10:40.020 --> 05:11:03.510
senate chamber: a validated model to generate a whole bunch of data, and then you can train a market model or whatever. And that's not unique. People have explored the idea. So so I think we really, that's part of what we will need to figure out is that. What are the various form factors, various requirements out there? And can we build an ecosystem that is

2010
05:11:03.510 --> 05:11:19.509
senate chamber: more flexible? That doesn't sort of target the cognitive psychology model. But that target a broader range? And again, can we make? Can we make the system flexible enough to do that? And that's part of the expansion that I think we need to do

2011
05:11:21.770 --> 05:11:22.710
senate chamber: a hetero.

2012
05:11:24.070 --> 05:11:42.909
Hedderik van Rijn: Yeah. So I think I also very much chime in with what Dario and then Andrea said, so, my my fellow director, when I get extremely enthusiastic about a new idea that would be great, and that all cognitive scientists would love. He always asks well, is this a vitamin or a painkiller.

2013
05:11:43.200 --> 05:11:58.059
Hedderik van Rijn: If we want to get others involved that are not into our sort of like nerdy nerdy, cognitive mindset. Many of the things that we do are vitamins so great if someone else does it. But I don't really need it.

2014
05:11:58.380 --> 05:12:05.160
Hedderik van Rijn: and I think that for the system to be more successful indeed, to many of these outside

2015
05:12:05.350 --> 05:12:18.800
Hedderik van Rijn: people. I think it's very important, like what Andrea said. What kind of what kind of ache did the system solve for them? So in what kind of pain were they that allowed them to use the system, and that if we want to grow.

2016
05:12:18.930 --> 05:12:20.800
Hedderik van Rijn: we should probably focus on that.

2017
05:12:20.920 --> 05:12:23.789
Hedderik van Rijn: Now, the other thing that I want to mention is that

2018
05:12:24.260 --> 05:12:26.080
Hedderik van Rijn: There are many people

2019
05:12:26.340 --> 05:12:32.449
Hedderik van Rijn: suggesting, or sort of like it. It came up quite a couple of times that we should have better repositories.

2020
05:12:33.412 --> 05:12:39.299
Hedderik van Rijn: I think that my 1st Xr workshop was in 98, 98,

2021
05:12:39.480 --> 05:12:46.440
Hedderik van Rijn: and I already recall that one of the big things there was that we should have an actr repository of models

2022
05:12:46.780 --> 05:12:57.939
Hedderik van Rijn: and any atr workshop that I've been at. We've mentioned this now, that could mean that we should finally do this, but it could also mean that it's clear that this is not the way to go.

2023
05:12:58.290 --> 05:13:06.580
Hedderik van Rijn: or that no one actually sees the need, and if you compare it to the sort of like what Mike brought up the the the our library.

2024
05:13:06.740 --> 05:13:17.129
Hedderik van Rijn: No one posts pardon. Osf, sort of like your separate code. No one posts the actual implementations of how these libraries are being used.

2025
05:13:17.350 --> 05:13:27.740
Hedderik van Rijn: And maybe that's just how it is. Maybe we should aim for sort of like small components that are easily embedded into the system, that that change some aspect of the system.

2026
05:13:28.510 --> 05:13:53.109
Hedderik van Rijn: But the actual model code just belongs in a way to the researcher, and you can get some inspiration. But it's not really the thing that you necessarily want or need to move. I don't want to work with Mike's Elmer models. I want to build my own. Even Andrea and I were collaborating now for months. We always rewrite the same models for ourselves. And I think this is just something that we probably need to live with.

2027
05:13:55.940 --> 05:13:56.790
senate chamber: I'm good.

2028
05:14:00.410 --> 05:14:02.870
Andrea Stocco: I I agree with that point.

2029
05:14:03.510 --> 05:14:15.300
Andrea Stocco: The issue of modularity also is something that keeps popping up a lot on the possibility of splitting the architecture into reusable components are a little bit easier to manage or embed or teach.

2030
05:14:15.560 --> 05:14:19.049
Andrea Stocco: One thing that I would like to to stress is that

2031
05:14:20.190 --> 05:14:25.020
Andrea Stocco: a big part of of an ecosystem.

2032
05:14:25.290 --> 05:14:47.719
Andrea Stocco: And this is what I got from talking to the few people who actually, I know have been involved in successful ecosystems, like the people involved in Spm. Afni, and what it's called psychopi is doing the things nobody wants to do, part of the Edic said, like nobody has ever built a repository because that takes time. Everybody in this room is happy to build

2033
05:14:47.990 --> 05:14:54.010
Andrea Stocco: free software. That's exciting. That's incredible. We're going to change the world. Nobody wants to do customer support.

2034
05:14:54.440 --> 05:15:15.169
Andrea Stocco: because that is terrible and boring, and we don't want to do it. Our students are going to do it, and so on. So part of the resources that need to be collected and invested is in finding out these boring jobs, maintaining repositories is not as exciting as writing software, but it needs to be done. Going around to do tutorials.

2035
05:15:15.640 --> 05:15:22.370
Andrea Stocco: Workshops is not exciting as giving presentations, but it needs to be done.

2036
05:15:22.490 --> 05:15:26.240
Andrea Stocco: and my few conversations with my colleague Rocham.

2037
05:15:26.400 --> 05:15:37.099
Andrea Stocco: The one thing that was very, very clear is that this 18 HA week he spent on this project his own open source software are 18 HA week that he hates.

2038
05:15:37.970 --> 05:15:41.400
Andrea Stocco: and they just need to be somewhat

2039
05:15:41.510 --> 05:15:49.030
Andrea Stocco: taking into consideration and in the development in the future development of architecture.

2040
05:15:49.530 --> 05:15:52.510
Andrea Stocco: and I don't have a good idea of

2041
05:15:52.620 --> 05:16:08.890
Andrea Stocco: how to handle this. But this is the part where some form of support monetary support becomes important because in the long run these things there is no other incentive than paying people to do this type of things, to answer emails about their bugs and so on.

2042
05:16:12.590 --> 05:16:16.259
senate chamber: I'll jump in with a comment to prerogative for one.

2043
05:16:17.290 --> 05:16:28.730
senate chamber: One of the issues that I run into is trying to get in trying to get the word out and say, related Dod projects. Because I kind of wear half a hat as academic half a hat is government

2044
05:16:28.900 --> 05:16:30.620
senate chamber: is.

2045
05:16:30.890 --> 05:16:40.309
senate chamber: I've been asked on and off for numerous years now I've talked to Christian Andre about it a little bit post cognitive modeling boot camps for

2046
05:16:40.580 --> 05:16:42.770
senate chamber: performers on various programs.

2047
05:16:46.030 --> 05:16:57.719
senate chamber: My expense for the funding agencies, which is not entirely a possible thing to do without funding.

2048
05:16:58.070 --> 05:17:03.260
senate chamber: Come to Hawaii, do a 1, do a 1 day workshop, but you have to organize everything and pay for yourself.

2049
05:17:03.780 --> 05:17:11.189
senate chamber: I I. So in this case, if if we do get to a community point where there is a

2050
05:17:14.150 --> 05:17:16.139
senate chamber: almost like a mobile summer school

2051
05:17:18.430 --> 05:17:26.500
senate chamber: that could be something that could be beneficial, where maybe we have some asynchronous learning

2052
05:17:26.970 --> 05:17:30.909
senate chamber: and something that could kind of help scaffold people

2053
05:17:31.880 --> 05:17:40.330
senate chamber: with the basic models I mean, the tutorials are great, but perhaps we need something that the greater world is.

2054
05:17:43.670 --> 05:17:44.861
senate chamber: can I make a suggestion.

2055
05:17:45.220 --> 05:17:51.290
senate chamber: So if I was wanting to get industry, I would try to figure out what actor can do better than anybody else.

2056
05:17:51.500 --> 05:17:54.500
senate chamber: and do it somehow, and make that

2057
05:17:55.713 --> 05:18:01.090
senate chamber: known. I guess one idea is that you guys know the arc competition.

2058
05:18:01.310 --> 05:18:09.659
senate chamber: I don't know what the status is now, but that's about 2 months ago the latest version, large language models. I think they were 8% or something. It's a hard task.

2059
05:18:10.020 --> 05:18:15.340
senate chamber: So solve that problem, hire anybody else, and then you won't have any problems, because people

2060
05:18:15.620 --> 05:18:22.290
senate chamber: at least you'll really be recognized like an outside suggestion is that crazy

2061
05:18:22.520 --> 05:18:27.119
senate chamber: makes sense. But I just thought of it. So you can just do more if you don't like it. But

2062
05:18:27.530 --> 05:18:34.970
senate chamber: from the outside it seems like you should try to solve that problem. Make it a competition. That's cheap kind of because you don't have to worry about customer service stuff.

2063
05:18:39.340 --> 05:18:57.609
senate chamber: I'd like to make a small comment coming from the Julia programming language community. I don't mean to throw another language into the ring, but I think there's a lot that we can learn from how Julia was made, because it was made for scientists where packages often lose the

2064
05:18:58.290 --> 05:19:02.517
senate chamber: the people that are founded to, or that created those packages. But

2065
05:19:03.150 --> 05:19:10.699
senate chamber: There's somehow always a way to to keep the models going. It might have to do with the language being

2066
05:19:11.820 --> 05:19:22.120
senate chamber: modular, but we do have an act, are tutorials, actor models, repository most of it by one single person.

2067
05:19:22.580 --> 05:19:28.019
senate chamber: There do exist things that are tested, continuously integrated.

2068
05:19:38.700 --> 05:19:41.210
senate chamber: There's a couple of comments in

2069
05:19:41.560 --> 05:19:43.419
senate chamber: chat, but I maybe wanted to

2070
05:19:45.070 --> 05:19:47.669
senate chamber: respond to a little bit because they're talking about.

2071
05:19:47.780 --> 05:19:49.410
senate chamber: and then go into Terry's work.

2072
05:19:51.020 --> 05:19:56.980
senate chamber: created python Axar and abandoned that project. He went on to reinvent Mendo in Python.

2073
05:19:58.850 --> 05:20:05.909
senate chamber: And is correct that to some extent the folks of our Waterloo

2074
05:20:06.490 --> 05:20:15.360
senate chamber: run a successful Mengo summer school annually currently has.

2075
05:20:20.110 --> 05:20:31.639
senate chamber: and it hasn't for several years. It is not compatible with Python 3 dot. 11, which is a couple years old. They have a company that's supposed to be maintaining.

2076
05:20:32.630 --> 05:20:35.740
senate chamber: using private sector funding. It is not

2077
05:20:35.880 --> 05:20:43.400
senate chamber: contributing updates to Ningo. So they have a similar problem to actar in that light.

2078
05:20:43.800 --> 05:20:50.950
senate chamber: I don't. They? They could probably give you a lot of explanation as to so what the problems are

2079
05:20:51.220 --> 05:20:54.960
senate chamber: with respect to finding maintainers

2080
05:20:55.120 --> 05:21:01.490
senate chamber: and getting getting at hand, software actively maintained. I don't think they can tell you the answers.

2081
05:21:03.690 --> 05:21:29.860
senate chamber: That's the 1st step to know what the problems are. There's a sense of a chicken and egg thing in that. We're talking about having summer schools to teach people to use the software. But we also need to make the software more accessible for summer schools. I wondered if I wanted to bring, like an undergrad, or like a master's student, into like maintaining Actar, or like creating like a

2082
05:21:30.540 --> 05:21:31.849
senate chamber: Api for it.

2083
05:21:32.130 --> 05:21:35.870
senate chamber: That would be more accessible.

2084
05:21:36.070 --> 05:21:40.030
senate chamber: I found myself wondering, like, what? What would I even say? That's like what's in it for them?

2085
05:21:42.063 --> 05:21:46.470
senate chamber: What like, academically speaking, like how you

2086
05:21:47.330 --> 05:21:53.720
senate chamber: write a thesis and advance an academic career out of. I wrote an Api for list factor.

2087
05:21:55.980 --> 05:21:57.990
senate chamber: I'm not sure.

2088
05:22:09.180 --> 05:22:11.060
senate chamber: Yeah, I'll I'll say I'm not either.

2089
05:22:11.700 --> 05:22:23.970
senate chamber: I think the I mean the incentivity. Like to incentivize somebody to do this, especially after the way Andrea characterized it is going to be very challenging, right? It's not a job anybody wants

2090
05:22:24.160 --> 05:22:27.800
senate chamber: 8 h, 8 h day he hates it.

2091
05:22:28.150 --> 05:22:36.750
senate chamber: And what what will my students get out of it? I mean for me, this is like the 900 pound gorilla in the room like, how do we?

2092
05:22:36.860 --> 05:22:37.990
senate chamber: How do we?

2093
05:22:38.410 --> 05:22:44.639
senate chamber: How do we do that? I don't know what? What is the incentive to get folks to maintain and contribute, and

2094
05:22:44.990 --> 05:22:48.380
senate chamber: I mean other than love of the architecture.

2095
05:22:48.540 --> 05:22:49.969
senate chamber: I'm not quite sure.

2096
05:22:54.090 --> 05:22:59.529
senate chamber: Money, I have something to say.

2097
05:22:59.920 --> 05:23:02.930
senate chamber: So th there was something like this.

2098
05:23:03.380 --> 05:23:33.000
senate chamber: I think. In my 3rd year of my undergrad we had a competition called my city's best scorer, all of us. We wanted to get jobs. So we judiciously participated in the competition, and the prize was for the top 10 winners to work on an internship that was unpaid. But just for experience purposes, so that it's on your resume. And you get to brag about, you know, working for a startup and things like that. So incentivizing is essentially, I think a competition

2099
05:23:33.120 --> 05:23:41.810
senate chamber: people can be made to believe that you're contributing to something good and just. Our classrooms on a website will boost a student's resumes, so

2100
05:23:41.940 --> 05:23:44.180
senate chamber: that should be enough, and then send it as well, I think.

2101
05:23:55.634 --> 05:24:02.370
senate chamber: have a question actually, and it is triggered by Mike Blurring.

2102
05:24:02.640 --> 05:24:08.150
senate chamber: who said that since 2,005. Approximately, the theory hasn't changed. Much.

2103
05:24:08.550 --> 05:24:13.760
senate chamber: So so for me, the theory is the foundation of everything else.

2104
05:24:14.280 --> 05:24:19.719
senate chamber: There are repositories. There are summer schools. There is software issues.

2105
05:24:20.750 --> 05:24:24.470
senate chamber: It all goes by the wayside in the theory.

2106
05:24:24.900 --> 05:24:29.719
senate chamber: doesn't, is not alive. It somehow falls behind the

2107
05:24:30.570 --> 05:24:35.350
senate chamber: the state of of the science more broadly construed.

2108
05:24:36.130 --> 05:24:42.299
senate chamber: And and maybe there is this one truth out there, and once you nail it, you're done. But

2109
05:24:42.870 --> 05:24:45.639
senate chamber: that was a view of science that is

2110
05:24:46.950 --> 05:24:49.719
senate chamber: as old as Newtonian mechanics and

2111
05:24:51.580 --> 05:24:56.170
senate chamber: physics has moved on since then. So what I'm getting at is.

2112
05:25:00.170 --> 05:25:07.800
senate chamber: how does the actor theory develop in the long run. Is there some sustainable model?

2113
05:25:08.211 --> 05:25:13.640
senate chamber: What is the thinking of that in the community right now? We kind of take it for granted

2114
05:25:14.360 --> 05:25:22.240
senate chamber: it came from one source, and everybody else was not everybody, but many people

2115
05:25:22.660 --> 05:25:28.709
senate chamber: or graduate students of John Anderson or postdocs of John Anderson, or somehow or other.

2116
05:25:29.810 --> 05:25:33.020
senate chamber: it is, there is a clear source of avatar.

2117
05:25:33.420 --> 05:25:38.899
senate chamber: But how? How do we go

2118
05:25:39.280 --> 05:25:47.779
senate chamber: into the long term future along those lines? And and how? How would that? This is what I struggle with.

2119
05:25:48.150 --> 05:26:00.619
senate chamber: but my main thought is, the theory is the foundation of everything else, and we discussed everything else. Nobody touched about how how theory can change?

2120
05:26:02.430 --> 05:26:19.779
senate chamber: Can I make a little amendment? To Mike's comment there has been theoretical problems since 2,005. You could say it's not been rapid enough, but there have been changes.

2121
05:26:20.560 --> 05:26:35.849
senate chamber: and some of them are reflected more or less from the core. Architecture which you download, and a number of those changes have taken the form of adding new modules to actar so

2122
05:26:36.600 --> 05:26:42.829
senate chamber: it can grow. And it's so. It's not.

2123
05:26:43.210 --> 05:26:46.800
senate chamber: It's not frozen in 2,005. In some sense.

2124
05:26:46.960 --> 05:26:55.190
senate chamber: What may be frozen in 2,005 is kind of the way of putting together the pieces rather than the pieces themselves, and

2125
05:26:55.470 --> 05:27:11.569
senate chamber: and I think maybe going back to our very 1st presentation, one can of the morning. One can think that maybe we should think about other ways of putting together the pieces as well. But the theory, at least, is not frozen. That's not quite right.

2126
05:27:12.630 --> 05:27:14.560
senate chamber: I my claim wasn't that it hadn't

2127
05:27:15.290 --> 05:27:22.520
senate chamber: haven't changed at all. But but the evolution of the theory was very rapid.

2128
05:27:22.920 --> 05:27:34.869
senate chamber: From the early nineties to the mid 2 thousands, and since then the number of software changes, the number of theory changes is much lower, but not 0. But there really was a transition around that timeline.

2129
05:27:37.751 --> 05:27:39.210
senate chamber: Nancy. I see you online.

2130
05:27:40.090 --> 05:27:49.990
Christopher Dancy: Thanks. So this is going back to previous conversation. Sorry, I know it's moved. And now, like that was my point. That was where my point was coming from.

2131
05:27:51.200 --> 05:28:01.488
Christopher Dancy: so this goes back to thinking about incentives. As I was thinking about this, you know part of where incentives come from. And I think we're actually kind of

2132
05:28:02.400 --> 05:28:06.649
Christopher Dancy: We're not done as well with having Frank have left. Given that he usually is one of the

2133
05:28:06.730 --> 05:28:28.430
Christopher Dancy: best self promoters of anything in the world, and I think but there's lesson to take from that right and naming things, and knowing how to make things count in different areas, right? And if we're talking about academia, it disincentivize the things we do, because as a community, we don't make it legible enough right? Because if we, if we want to think of ourselves as a community.

2134
05:28:28.430 --> 05:28:39.899
Christopher Dancy: communities, make things legible, Llms and the ways that those things are marketed in the academic realm, right become legible, and and the things you do with them become legible because of the communities around them.

2135
05:28:40.317 --> 05:28:42.613
Christopher Dancy: And so I think that's that's 1 point

2136
05:28:43.130 --> 05:28:46.229
Christopher Dancy: and relate to that, like, even think about models, right? Model repositories.

2137
05:28:46.470 --> 05:29:05.639
Christopher Dancy: Sure, we create the system. But we don't. There. People just don't publish it. There's not really that open right? Their models aren't there? Like if you can still upload them. Theoretically, people won't do it because they don't want to do it, and and some of it is because of how hard it is. Some of it is because I'm not sure everybody wants to release their models. At times. And so there, there's a

2138
05:29:06.240 --> 05:29:31.473
Christopher Dancy: I think, as a community there, there are things and culpability that we have to own, and that we haven't encouraged it in the best ways as a community. And and I think thinking about that, in addition to the creating, the architecture and the pipelines themselves. We have to think about down to what we require, even at conferences in in terms of publishing things like that. Are we gonna require models or data, or something like that at times.

2139
05:29:32.030 --> 05:29:38.390
Christopher Dancy: certainly we. We encourage it and reviews will encourage it. But it's not explicit. So this is just kind of a point on

2140
05:29:38.550 --> 05:29:41.349
Christopher Dancy: when we think about incentivizing. Some of it is institutional.

2141
05:29:41.490 --> 05:29:57.310
Christopher Dancy: but there is an also, like, you know, a macro level. But there also is this Meso level of community that we control collectively. And and so I think we have to remember that and and think about that as well as we think about how we're creating this full ecosystem and the importance of community values.

2142
05:30:03.240 --> 05:30:04.280
senate chamber: Okay, yeah.

2143
05:30:06.080 --> 05:30:15.230
senate chamber: Just a follow up thing to what Dr. Dancy said. I think it's also pointed out.

2144
05:30:15.560 --> 05:30:22.110
senate chamber: The same thing has been discussed over several conferences, but not has been done.

2145
05:30:23.490 --> 05:30:48.490
senate chamber: one thing I would like to borrow from software terminology is the thing called action items from retrospective meeting, which I think this is so. If somebody is in charge of just having all the questions we discuss in one place, and maybe some pointers to possible answers, I think, it will help for people to reach out, slash, contribute to the question that we are discussing and to see something.

2146
05:30:53.080 --> 05:31:18.660
senate chamber: I want to make a small comment as an external viewer of this discussion, as someone who doesn't currently use actar, but as someone who has always heard of it and never learned it. To me as a naive observer it seems that the best way to learn actar is to be part of a lab where people are actively using it already, and in terms of bringing new people or motivating new people. To want to contribute

2147
05:31:19.000 --> 05:31:29.680
senate chamber: can be a bit of a barrier to entry. So just following on earlier comments about these various modules and knowledge being a bit hidden, I think it may be helpful to look more into breaking this barrier.

2148
05:31:38.130 --> 05:31:45.209
senate chamber: I think 2 things that would be very helpful from a student perspective as someone who's still kind of learning their way through Aptar.

2149
05:31:46.140 --> 05:31:52.400
senate chamber: One is kind of Youtube tutorials. We don't have to do a summer camp. Youtube tutorials are accessible all the time.

2150
05:31:52.530 --> 05:32:12.500
senate chamber: but it does raise the question of who is going to make the Youtube tutorials. Another thing is kind of the equivalent of a stack overflow for actar modules. When I need help. There is a very specific number of people that I know that I can ask for help. But if I could post in a forum and say, Hey, this is a problem I'm having.

2151
05:32:12.900 --> 05:32:18.670
senate chamber: If someone else is working on that on the other side of the world that would be great if they could chime in and

2152
05:32:19.160 --> 05:32:23.390
senate chamber: kind of have a discussion that could be accessed by everyone in this room.

2153
05:32:24.310 --> 05:32:30.559
senate chamber: Then that would be, I think, a very helpful way to and keep that discussion going.

2154
05:32:31.630 --> 05:32:36.249
senate chamber: Yeah, I think a subreddit also in my head.

2155
05:32:36.430 --> 05:32:41.350
senate chamber: Just communicate along with us, stacked our overflow.

2156
05:32:43.980 --> 05:32:46.950
senate chamber: I mean, we do have the active users

2157
05:32:47.150 --> 05:32:51.820
senate chamber: list, which does serve does kind of serve that purpose if you're on it.

2158
05:32:52.080 --> 05:32:57.030
senate chamber: But you know, as students who find it in my computer, I also agree with that.

2159
05:32:57.880 --> 05:32:58.620
senate chamber: Yeah.

2160
05:32:59.200 --> 05:33:11.381
senate chamber: yeah, just wanted actually to mention Terry, who is now virtually there. And we could ask him about self sustained ecosystems, and maybe also about how

2161
05:33:12.510 --> 05:33:14.670
senate chamber: models could be

2162
05:33:15.000 --> 05:33:22.809
senate chamber: put into githubs or changed, or to to have more wide global system. So maybe, Terry, you could

2163
05:33:23.430 --> 05:33:25.949
senate chamber: have some thoughts on on that.

2164
05:33:27.610 --> 05:33:32.260
Terry Stewart (NRC; he/him): Is sure I can wander into the conversation after

2165
05:33:32.420 --> 05:33:51.130
Terry Stewart (NRC; he/him): not seeing much of the earlier stuff. Good to see everyone. By the way, thank you, Nelly, for reaching out. I am looking forward to being more connected with the community in the next little while. I I kind of burned out for a little while. But that said

2166
05:33:52.230 --> 05:34:00.149
Terry Stewart (NRC; he/him): The one thing like I I echo everything people are are saying here, and one thing that I'm

2167
05:34:00.580 --> 05:34:04.379
Terry Stewart (NRC; he/him): attempting to work on here is

2168
05:34:04.975 --> 05:34:20.000
Terry Stewart (NRC; he/him): online, like versions of models that are not just like posted to Github or something like that. But like online, interactive versions embedded in a web page where you don't even have to install anything. And that's

2169
05:34:21.130 --> 05:34:29.830
Terry Stewart (NRC; he/him): but I think with all of these solutions we end up getting. I think this is huge question of whose job it is to maintain this. Whose job is it to do it?

2170
05:34:31.235 --> 05:34:32.520
Terry Stewart (NRC; he/him): How?

2171
05:34:32.760 --> 05:34:39.309
Terry Stewart (NRC; he/him): Yeah, how how we can in encourage that but I would love to

2172
05:34:40.270 --> 05:34:48.727
Terry Stewart (NRC; he/him): have a group of people from this community brainstorming and trying things out, and maybe a small working group in that in that direction.

2173
05:34:49.380 --> 05:35:10.300
Terry Stewart (NRC; he/him): just anything to make these things more accessible and immediate, and to drop that barrier of entry, because I think, like cognitive models are so exciting right now. But I spend so much time in the Llm. Community right now, and they they haven't even taken psych 101. They do not understand cognition.

2174
05:35:11.040 --> 05:35:14.180
Terry Stewart (NRC; he/him): and they're missing everything that this community has done.

2175
05:35:14.300 --> 05:35:24.334
Terry Stewart (NRC; he/him): And I really want to. How do we make this accessible? How do we spread that word? How do we show all of that? So

2176
05:35:24.860 --> 05:35:38.750
Terry Stewart (NRC; he/him): yeah. The particular. The only new thing I have to say in all of that is, I do think we have the potential of having live interactive versions. That could be something that we can do that. Maybe it's a little bit harder in the Lm space.

2177
05:35:39.922 --> 05:35:46.735
Terry Stewart (NRC; he/him): but yeah, so I'll I'll leave it at that. But good to see everyone.

2178
05:35:50.390 --> 05:35:51.410
Terry Stewart (NRC; he/him): Wow!

2179
05:35:51.770 --> 05:35:56.199
senate chamber: So I guess to go back to the point of forms and online stuff that

2180
05:35:56.770 --> 05:35:59.869
senate chamber: we have tried that in the past. I will say probably about

2181
05:36:01.040 --> 05:36:28.680
senate chamber: it's hard for me to judge how long ago, but at 1 point we had somebody that said we should have a forum on the website, and we put up a forum on the website. And the only person answered questions there was me. So we've got to get the community involved. So even if we put something like that, we do have that barrier of trying to get the community to also support it. So I understand it's useful for people new to the community. But we've got to get the

2182
05:36:28.730 --> 05:36:32.399
senate chamber: support from people in the community to help that, too.

2183
05:36:33.990 --> 05:36:39.930
senate chamber: You will notice it's not there anymore. So that did kind of go away at 1 point. But

2184
05:36:40.080 --> 05:36:44.690
senate chamber: I definitely that could be useful if we can get the support.

2185
05:36:46.360 --> 05:36:49.830
senate chamber: Yeah, I think for conferences. We are like

2186
05:36:49.950 --> 05:37:12.530
senate chamber: early career, like 1st year Pac students. Or you know, early master students do not come to conferences because we only come when we're presenting something which means we are far too much into actor. So an online, yeah, tutorial kind of thing. For the people who want to learn Acta, I think, will help Acta. Workshop and conferences might not

2187
05:37:12.760 --> 05:37:17.119
senate chamber: help the students who want to come, but are not coming due to lack of resources.

2188
05:37:19.250 --> 05:37:19.650
senate chamber: No.

2189
05:37:23.040 --> 05:37:25.050
senate chamber: Or is it? Terry had his hand up.

2190
05:37:25.050 --> 05:37:44.780
Terry Stewart (NRC; he/him): Yep. Oh, yeah, just to echo with that as well. We had a similar thing happening with the Nengo community, the sort of the neural modeling thing. We had this really nice forum tons of discussion, tons of stuff going on. But that only happened while it was someone's job to answer questions.

2191
05:37:45.141 --> 05:37:56.890
Terry Stewart (NRC; he/him): and whatnot. And so for a while that software package was being developed by a company, a spin off company from here. But then eventually that company decided that wasn't the direction they were gonna go. And so they

2192
05:37:56.960 --> 05:38:03.409
Terry Stewart (NRC; he/him): stopped answering things on the Forum and the community just immediately died.

2193
05:38:03.500 --> 05:38:12.280
Terry Stewart (NRC; he/him): And so yeah, but it it thrived and was great during the time where it was someone's job to do that. And I don't. I don't know.

2194
05:38:12.870 --> 05:38:13.470
Terry Stewart (NRC; he/him): Yeah,

2195
05:38:18.160 --> 05:38:19.470
senate chamber: Can I see questions of the

2196
05:38:19.600 --> 05:38:21.880
senate chamber: younger people in the room students?

2197
05:38:23.320 --> 05:38:29.269
senate chamber: like to me trying to get people some part of the pose projects. This is a pose idea, a question. So

2198
05:38:29.410 --> 05:38:34.420
senate chamber: getting people to come, it's like, I'm a musician. Get people to come to a gig that I play.

2199
05:38:34.480 --> 05:39:01.299
senate chamber: you know, if you play a bar where there's a lot of people you don't have to worry about getting people to come there. They come to the bar. If you've been around a long time, and you play things that no one else plays, or you play them better than anybody else. People come to, no matter where you go. But if you're just trying to, you know you play, and you have a small group of your friends who like the music, and you go to another city, or you go to another bar in your same town. No one's going to show up. The bar isn't already have its own crowd. I think that's part of the problem proposed, which is.

2200
05:39:01.820 --> 05:39:20.780
senate chamber: you know, I did mention the art challenge. But for young people who are learning things, the discussion around, how do we get them? Make it easier for them to learn? I think that's the wrong emphasis. The emphasis is, how do we make them want to learn so bad that we don't have to worry about putting online videos on. I mean, I think that's the wrong approach.

2201
05:39:25.560 --> 05:39:26.280
senate chamber: right?

2202
05:39:28.820 --> 05:39:36.611
senate chamber: This is Paris. But I have a comment from Frank Reser.

2203
05:39:37.780 --> 05:39:39.482
senate chamber: Tell him no.

2204
05:39:40.840 --> 05:39:53.539
senate chamber: Is it chicken chicken? But he's he suggests that maybe we can have a quarterly actual meet up in Zoom.

2205
05:39:57.180 --> 05:39:59.379
senate chamber: I would prefer the wife.

2206
05:40:01.930 --> 05:40:15.520
senate chamber: May I talk a little bit about my my strategy? My strategy is that of a scientific facilitator. Right. I'm an engineer, and I try to facilitate the technology that's available as a way of teaching. And so I,

2207
05:40:15.860 --> 05:40:19.869
senate chamber: by default, have a group of students who want to learn.

2208
05:40:20.550 --> 05:40:25.800
senate chamber: and I can offer them what I can teach. But code is my way of teaching.

2209
05:40:25.910 --> 05:40:38.039
senate chamber: and I think this is, I see there a huge potential for people who not only maintain, learn to use the code, but also learn to maintain the code as they learn the architecture.

2210
05:40:38.600 --> 05:40:42.309
senate chamber: Personal strategy. It's not being implemented in more than 6 people.

2211
05:40:48.510 --> 05:40:53.970
senate chamber: Maybe it would be a good idea to bring the different aspects together that we just discussed

2212
05:40:55.020 --> 05:41:00.100
senate chamber: to bring young researchers onto the conference. We could

2213
05:41:00.190 --> 05:41:26.780
senate chamber: have like a special session or something, and offer the available models and environments where there's already material to get started with modeling to add something new in there that is not too much work, but has a new, so something to present, and so bring people on the conference and also start. I think it could also be something like a growing process

2214
05:41:26.860 --> 05:41:43.019
senate chamber: to start, use models that are already there and environments, and to to really have a benefit from previous work and and use that. So maybe that would be something very helpful for different purposes.

2215
05:41:43.630 --> 05:41:47.159
senate chamber: And I think that would be easy to implement.

2216
05:41:51.510 --> 05:41:54.889
senate chamber: So I think there's something to be said about

2217
05:41:55.300 --> 05:41:58.650
senate chamber: as a student. If the lab you're in isn't using actor.

2218
05:41:58.930 --> 05:42:04.510
senate chamber: you might work on it. But like where you're getting the support from. So our.

2219
05:42:05.510 --> 05:42:08.090
senate chamber: So I have a potential solution. What if

2220
05:42:08.720 --> 05:42:15.240
senate chamber: there was like? There were collaboration opportunities for students to collaborate with other people at universities on a project.

2221
05:42:15.580 --> 05:42:20.319
senate chamber: so that then they have that support. They can play a part in it. They can work on a piece of it.

2222
05:42:21.482 --> 05:42:26.747
senate chamber: That might be one option, and then they can ask questions from that. So it's sort of like

2223
05:42:28.270 --> 05:42:32.150
senate chamber: being part of another lab that you get that kind of support from?

2224
05:42:42.340 --> 05:42:43.620
senate chamber: Not sure. Yeah.

2225
05:42:44.610 --> 05:42:46.469
senate chamber: Very smart suggestion.

2226
05:42:47.390 --> 05:42:53.370
senate chamber: One is to help an octar community like in discord. For example, I'll just discord

2227
05:42:53.860 --> 05:42:59.210
senate chamber: and the discord community. Maybe we can have the like.

2228
05:42:59.560 --> 05:43:03.209
senate chamber: at least the blogs that are actually working on Arthur. And then

2229
05:43:03.390 --> 05:43:06.569
senate chamber: those list of logs are not computers.

2230
05:43:06.730 --> 05:43:08.890
senate chamber: So that way we can.

2231
05:43:10.280 --> 05:43:13.899
senate chamber: Alex's idea of like collaboration work.

2232
05:43:14.100 --> 05:43:18.609
senate chamber: And also on top of that, we can have, for example, a website where

2233
05:43:18.800 --> 05:43:38.350
senate chamber: before that on the discord we can have access to you guys. That means whenever we want to create resources. For example, here's the students, I feel like we're so open to create the resources like the Youtube resources. And that way, you guys can review whatever we're going to create.

2234
05:43:38.460 --> 05:43:46.749
senate chamber: I know you guys are very busy, but you can just not see. Oh, you're missing this type of detail from the Avatar.

2235
05:43:48.160 --> 05:44:07.630
senate chamber: And on top of that we can have a website of, for example, we can call it Icm labs. We already have that on the survey where in the Icm labs click on community, we can have open source models from Aktar. Not just Aktar, but other like, Icm

2236
05:44:07.900 --> 05:44:12.739
senate chamber: research, research, you know, yeah, projects that we were.

2237
05:44:21.110 --> 05:44:26.239
senate chamber: So actually, I know we talked about on the website

2238
05:44:29.080 --> 05:44:32.910
senate chamber: names of people that are involved in the community. Maybe some stuff they do where they work.

2239
05:44:33.420 --> 05:44:40.200
senate chamber: and we can even include something like willing to collaborate, I would say, like, Take someone on. But

2240
05:44:40.510 --> 05:44:44.850
senate chamber: everyone, you know, a lot of people working on different projects. And someone could

2241
05:44:45.150 --> 05:44:46.960
senate chamber: do something in parallel with that

2242
05:44:47.580 --> 05:44:49.139
senate chamber: that might be a way to

2243
05:44:49.250 --> 05:44:54.359
senate chamber: see someone that's working on something that you're interested in maybe reach out to them. See if there's an opportunity to

2244
05:44:54.860 --> 05:45:00.389
senate chamber: learn a little bit from the person. Or maybe, like I said, play a little role in a project that's ongoing.

2245
05:45:10.140 --> 05:45:12.479
senate chamber: Okay? So I'm not young. But I'm a student.

2246
05:45:13.470 --> 05:45:18.230
senate chamber: And I think what Alex said is really great is that exposure

2247
05:45:18.520 --> 05:45:25.820
senate chamber: to someone who is actually doing the modeling, the exposure to others

2248
05:45:26.300 --> 05:45:33.599
senate chamber: who present their work, so that I think being able to come to conferences

2249
05:45:34.800 --> 05:45:45.830
senate chamber: can be play a big part in that. The challenge, like being challenged by my advisor to create something that's going to be presented at a conference. So

2250
05:45:46.030 --> 05:45:55.980
senate chamber: there's the challenge is important. The exposure at the conference, the recognition that

2251
05:45:56.290 --> 05:46:00.019
senate chamber: I accomplished something, and I learned something in the process.

2252
05:46:01.840 --> 05:46:05.669
senate chamber: So I really think that the opportunity with the

2253
05:46:06.380 --> 05:46:12.799
senate chamber: something at the conference like not just presented paper, but maybe some special resources for students.

2254
05:46:13.010 --> 05:46:18.489
senate chamber: because students don't have as many resources to just come to a conference unless they're presenting.

2255
05:46:18.760 --> 05:46:29.489
senate chamber: But if there were some like, there's a a special discount for training members that's helpful. But maybe

2256
05:46:30.350 --> 05:46:33.820
senate chamber: one or 2 scholarships that can be provided

2257
05:46:33.950 --> 05:46:38.599
senate chamber: to allow the student to be exposed to other people's work.

2258
05:46:43.180 --> 05:46:44.845
senate chamber: Virtual office hours.

2259
05:46:53.140 --> 05:46:53.949
senate chamber: are we?

2260
05:46:54.140 --> 05:46:55.729
senate chamber: Mexico? Do it? One more.

2261
05:46:55.920 --> 05:46:59.690
senate chamber: Yeah, just to, I guess.

2262
05:47:00.020 --> 05:47:01.799
senate chamber: Briefly add a couple of things.

2263
05:47:02.910 --> 05:47:06.940
senate chamber: There's.

2264
05:47:07.050 --> 05:47:22.290
senate chamber: I think, a significant way to go before actar is going to be accessible. And before it's going to be something that students really really want to use. But there's there's a couple somewhat

2265
05:47:22.400 --> 05:47:26.960
senate chamber: clear steps to to take before that point.

2266
05:47:27.130 --> 05:47:38.699
senate chamber: One of them might be putting an Ssl. Certificate on the Aptar homepage, so it doesn't look like a sketchy website. I have been trying to get support. To get that fixed

2267
05:47:38.960 --> 05:47:40.840
senate chamber: and get working through

2268
05:47:40.950 --> 05:47:52.529
senate chamber: Cmu support is not necessarily the easiest thing. It is shocking to me that Cmu support would be slow and antiquated. That does not seem. You would think that

2269
05:47:55.440 --> 05:48:14.290
senate chamber: But besides that, maybe to steal some language. It might be a good idea to propose a couple of action items for next year, and one of those would just be migrating to git and setting up a public repository that anybody can see, and

2270
05:48:15.010 --> 05:48:25.350
senate chamber: making a fair amount of documentation in a standard format that's available that is used by other software libraries

2271
05:48:26.074 --> 05:48:41.829
senate chamber: as available and accessible as possible. And that's well indexed by search engines would be, I think, a really good start if your goal is to eventually motivate people to contribute. So the very baseline has to be like somebody who

2272
05:48:41.950 --> 05:48:43.610
senate chamber: isn't already like

2273
05:48:43.760 --> 05:48:49.719
senate chamber: on the mailing list, and able to like ask people questions in the sort of old style way we used to do software maintenance.

2274
05:48:52.250 --> 05:49:00.360
senate chamber: having it be accessible to a newer generation of of come up programming lazily.

2275
05:49:01.640 --> 05:49:08.580
senate chamber: would at least be a starting place. If you want to, then move on to making it something that they really want to participate in.

2276
05:49:15.380 --> 05:49:19.669
senate chamber: So I guess, on the documentation point. Do you mean using actar or

2277
05:49:19.770 --> 05:49:33.029
senate chamber: working with the soft? Because there's 2 points right? There's the the modeling side of that. And the implementation side. So which side do you? Are you saying both? I guess I mean well, well, kind of I mean. So so

2278
05:49:33.220 --> 05:49:37.580
senate chamber: part of it is and

2279
05:49:42.035 --> 05:49:43.775
senate chamber: part of it is

2280
05:49:47.030 --> 05:49:48.200
senate chamber: When

2281
05:49:48.530 --> 05:50:00.799
senate chamber: I'm gonna I'm gonna pick on Terry for a second and use Ningo as an example. I don't know if he's still here. But when I have a problem in Dingo.

2282
05:50:01.030 --> 05:50:06.809
senate chamber: what I do is I open up. The men, go source code, and I look at the source code. And I look at how it works.

2283
05:50:07.130 --> 05:50:09.780
senate chamber: Figure out how it works. And once I figured out how it works.

2284
05:50:09.970 --> 05:50:10.900
senate chamber: I think so.

2285
05:50:11.320 --> 05:50:20.400
senate chamber: And sometimes that's very easy to do. Sometimes it is quite circuitous, because the design of it is somewhat circuitous.

2286
05:50:23.460 --> 05:50:28.860
senate chamber: But the the at the very least right. The Mingo Source code is

2287
05:50:29.070 --> 05:50:34.750
senate chamber: fairly clearly written and easy to access publicly and search through

2288
05:50:35.140 --> 05:50:43.470
senate chamber: etc. However, the documentation for Mengo is not great.

2289
05:50:44.040 --> 05:50:48.990
senate chamber: It's very incomplete in terms of

2290
05:50:51.260 --> 05:50:55.309
senate chamber: in terms of if I want to build Mingo models.

2291
05:50:56.598 --> 05:51:12.389
senate chamber: What I need is a complete Api. That tells me exactly how these objects will behave without having to look at the source code or completely, without having to look at the source code. I don't think that exists for actor right now.

2292
05:51:14.260 --> 05:51:26.079
senate chamber: but then, yeah, from from a software maintenance perspective. Obviously, you want well documented code and like a good model for like contribution. But I think the more important thing is, if you want people to work on, or with actar

2293
05:51:28.480 --> 05:51:30.869
senate chamber: from like starting from starting from a

2294
05:51:31.720 --> 05:51:35.250
senate chamber: being very unfamiliar and and not having a whole lot of people to talk to.

2295
05:51:35.979 --> 05:51:41.250
senate chamber: The the thing that you need is a very well documented Api.

2296
05:51:41.850 --> 05:51:55.439
senate chamber: So I will say, there's a 600 page reference or 500 page reference manual available with actar. I'm familiar with the reference manual. Okay, it's sort of different model of document. You're right.

2297
05:51:56.140 --> 05:52:03.039
senate chamber: but it's a it's a kind of a different model of documentation than, and and part of part of this is working with

2298
05:52:04.140 --> 05:52:13.280
senate chamber: undergraduates and early grad students and walking them through documentation.

2299
05:52:14.440 --> 05:52:15.230
senate chamber: But,

2300
05:52:19.080 --> 05:52:27.189
senate chamber: The the formatting of the documentation is something that matters a lot and

2301
05:52:28.740 --> 05:52:35.149
senate chamber: a manual is great. However, the phrase Rtfm exists for a reason. That's because people don't.

2302
05:52:37.720 --> 05:52:41.379
senate chamber: But they should, they should they should.

2303
05:52:43.370 --> 05:52:47.279
senate chamber: But if if you're if if the problem is like people won't jump in.

2304
05:52:47.540 --> 05:52:52.760
senate chamber: Then expecting them to read the manual before jumping in, and not even bothering to read the manual and want them to jump in is not

2305
05:52:53.530 --> 05:52:54.630
senate chamber: good place to start.

2306
05:52:57.410 --> 05:53:06.759
senate chamber: Do we want to give, since it was his talk last last question, absolutely last comment, and then.

2307
05:53:09.530 --> 05:53:11.500
Andrea Stocco: Yeah, thank you. I I

2308
05:53:12.100 --> 05:53:28.250
Andrea Stocco: I don't want to be necessarily closing out. But I think that my comment could be a good closing point. So this entire discussion has been incredible, and literally my wrist hurts because I haven't taken notes since I was in graduate school. So

2309
05:53:28.910 --> 05:53:45.519
Andrea Stocco: I'm going to sue you all for carpartana after this. But jokes aside one of the things that I would like to stress, that is one of the things that we learned in the process of writing. This proposal is that every model that exists, and every model that has been suggested comes with this

2310
05:53:45.700 --> 05:53:52.329
Andrea Stocco: negative consequences. Part of the idea of an ecosystem is that it's being an ecosystem is alive.

2311
05:53:52.840 --> 05:53:59.720
Andrea Stocco: And I think that our main goal right now is not to create a perfect self-supporting system, but just move

2312
05:53:59.920 --> 05:54:06.940
Andrea Stocco: incrementally towards it, and start to think about the problems we might face and what are the 1st incremental steps to take.

2313
05:54:07.100 --> 05:54:14.059
Andrea Stocco: And I think that this discussion so far has been excellent in giving us a great 1st directions.

2314
05:54:19.030 --> 05:54:35.930
senate chamber: And as Andrea mentioned, this is the start of a process, and that process involved reaching out to a lot of people and doing interviews and figuring out their needs and their suggestions, and what the ecosystem from their perspective looks like.

2315
05:54:36.030 --> 05:54:55.720
senate chamber: So we're definitely in the weeks and months coming, doing a lot of that reaching out. But if you, if you want to contribute and you're afraid we would miss you, please send us email Andrea myself then, and we'll we'll make sure we reach out and we we make you part of the process.

2316
05:54:56.030 --> 05:55:20.100
senate chamber: So thank you all for coming. I thought it was a really successful workshop, one of the best attended in many years. So again, if you have any feedback on sort of the format and all that was a bit different this year, and I think it worked well. But you know, just like it's all part of the ecosystem, too, the workshop. So thank you very much, and I hope you have a

2317
05:55:20.100 --> 05:55:21.209
senate chamber: thank you. Christian.

2318
05:55:25.410 --> 05:55:26.130
senate chamber: Okay.